[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:00:47 dist-test-slave-sg4j krb5kdc[3715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:00:47 dist-test-slave-sg4j krb5kdc[3715](info): set up 2 sockets
Sep 27 22:00:47 dist-test-slave-sg4j krb5kdc[3715](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 22:00:48.443624 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.009s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:00:48 dist-test-slave-sg4j krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:00:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:00:48.660218 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:35275
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:35275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:00:49.105746 3728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:00:49.106185 3728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:00:49.106313 3728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:00:49.115931 3728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:00:49.116106 3728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:00:49.116161 3728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:00:49.116228 3728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:00:49.116273 3728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:00:49.129770 3728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:35275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:00:49.131608 3728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:00:49.133502 3728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:00:49.149590 3734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:49.150102 3736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:49.150144 3733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:00:49.151474 3728 server_base.cc:1034] running on GCE node
I20260927 22:00:49.152696 3728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:00:49.154749 3728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:00:49.156132 3728 hybrid_clock.cc:648] HybridClock initialized: now 1790546449156023 us; error 107 us; skew 500 ppm
I20260927 22:00:49.156654 3728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:00:49 dist-test-slave-sg4j krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546449, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:00:49.160948 3728 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:00:49.163394 3728 webserver.cc:460] Webserver started at http://127.3.159.62:43343/ using document root <none> and password file <none>
I20260927 22:00:49.164398 3728 fs_manager.cc:359] Metadata directory not provided
I20260927 22:00:49.164579 3728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:00:49.164983 3728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:00:49.169293 3728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "b4d66a21ef5c4b2ea59996ef2d46529e"
format_stamp: "Formatted at 2026-09-27 22:00:49 on dist-test-slave-sg4j"
server_key: "f708a2b77d25fe1b273a341f1e647781"
server_key_iv: "fdbf61e2bf559a493858c95bba4ec3ff"
server_key_version: "encryptionkey@0"
I20260927 22:00:49.170516 3728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "b4d66a21ef5c4b2ea59996ef2d46529e"
format_stamp: "Formatted at 2026-09-27 22:00:49 on dist-test-slave-sg4j"
server_key: "f708a2b77d25fe1b273a341f1e647781"
server_key_iv: "fdbf61e2bf559a493858c95bba4ec3ff"
server_key_version: "encryptionkey@0"
I20260927 22:00:49.177932 3728 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:00:49.183254 3743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:49.185658 3728 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:00:49.185988 3728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "b4d66a21ef5c4b2ea59996ef2d46529e"
format_stamp: "Formatted at 2026-09-27 22:00:49 on dist-test-slave-sg4j"
server_key: "f708a2b77d25fe1b273a341f1e647781"
server_key_iv: "fdbf61e2bf559a493858c95bba4ec3ff"
server_key_version: "encryptionkey@0"
I20260927 22:00:49.186327 3728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:00:49.210028 3728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:00:49.216712 3728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:00:49.217113 3728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:00:49.247313 3728 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:35275
I20260927 22:00:49.247457 3794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:35275 every 8 connection(s)
I20260927 22:00:49.249476 3728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:00:49.255267 3795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:00:49.255869 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 3728
I20260927 22:00:49.256234 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:00:49.258658 3708 external_mini_cluster.cc:1428] Setting key dd22889d570fd4310d101e35344e5dab
I20260927 22:00:49.270722 3795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e: Bootstrap starting.
I20260927 22:00:49.274305 3795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e: Neither blocks nor log segments found. Creating new log.
I20260927 22:00:49.275909 3795 log.cc:824] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:00:49.279601 3795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e: No bootstrap required, opened a new log
I20260927 22:00:49.289456 3795 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d66a21ef5c4b2ea59996ef2d46529e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35275 } }
I20260927 22:00:49.289956 3795 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:00:49.290073 3795 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d66a21ef5c4b2ea59996ef2d46529e, State: Initialized, Role: FOLLOWER
I20260927 22:00:49.290944 3795 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d66a21ef5c4b2ea59996ef2d46529e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35275 } }
I20260927 22:00:49.291312 3795 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:00:49.291565 3795 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:00:49.291795 3795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:00:49.294886 3795 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d66a21ef5c4b2ea59996ef2d46529e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35275 } }
I20260927 22:00:49.295661 3795 leader_election.cc:302] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4d66a21ef5c4b2ea59996ef2d46529e; no voters:
I20260927 22:00:49.296419 3795 leader_election.cc:288] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:00:49.296751 3800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:00:49.298205 3800 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [term 1 LEADER]: Becoming Leader. State: Replica: b4d66a21ef5c4b2ea59996ef2d46529e, State: Running, Role: LEADER
I20260927 22:00:49.299429 3800 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d66a21ef5c4b2ea59996ef2d46529e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35275 } }
I20260927 22:00:49.299773 3795 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 22:00:49 dist-test-slave-sg4j krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:00:49.304667 3801 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4d66a21ef5c4b2ea59996ef2d46529e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d66a21ef5c4b2ea59996ef2d46529e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35275 } } }
I20260927 22:00:49.305580 3801 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [sys.catalog]: This master's current role is: LEADER
I20260927 22:00:49.304726 3802 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4d66a21ef5c4b2ea59996ef2d46529e. Latest consensus state: current_term: 1 leader_uuid: "b4d66a21ef5c4b2ea59996ef2d46529e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d66a21ef5c4b2ea59996ef2d46529e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35275 } } }
I20260927 22:00:49.305774 3802 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e [sys.catalog]: This master's current role is: LEADER
I20260927 22:00:49.307118 3806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:00:49.321545 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:49.264290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45440 (local address 127.3.159.62:35275)
0927 22:00:49.265518 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:49.265544 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:49.267239 (+ 1695us) server_negotiation.cc:407] Connection header received
0927 22:00:49.279613 (+ 12374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:49.279653 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:49.280407 (+ 754us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:49.281049 (+ 642us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:49.285512 (+ 4463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:49.287510 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:49.290273 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:49.291715 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:49.292017 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:49.302508 (+ 10491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:49.302529 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:49.302594 (+ 65us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:49.302684 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:49.309099 (+ 6415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:49.310897 (+ 1798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:49.310916 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:49.310932 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:49.311392 (+ 460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:49.312363 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:49.312372 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:49.312379 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:49.313623 (+ 1244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:49.318265 (+ 4642us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:49.318785 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 22:00:49.319495 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":504,"threads_started":1}
I20260927 22:00:49.332634 3806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:00:49.345709 3806 catalog_manager.cc:1347] Generated new cluster ID: c9ac9fa35bbd45038f4c83faf11dfbda
I20260927 22:00:49.345878 3806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:00:49.361027 3806 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:00:49.363698 3806 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:00:49.376627 3806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4d66a21ef5c4b2ea59996ef2d46529e: Generated new TSK 0
I20260927 22:00:49.378376 3806 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:00:49.440048 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35275
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:00:49.873160 3821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:00:49.873600 3821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:00:49.873757 3821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:00:49.882570 3821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:00:49.882740 3821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:00:49.882941 3821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:00:49.895550 3821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35275
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:00:49.897469 3821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:00:49.899205 3821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:00:49.917062 3827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:49.917025 3829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:49.917817 3826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:00:49.919116 3821 server_base.cc:1034] running on GCE node
I20260927 22:00:49.919879 3821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:00:49.921011 3821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:00:49.922288 3821 hybrid_clock.cc:648] HybridClock initialized: now 1790546449922187 us; error 83 us; skew 500 ppm
I20260927 22:00:49.922732 3821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:00:49 dist-test-slave-sg4j krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546449, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:00:49.926796 3821 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:00:49.928954 3821 webserver.cc:460] Webserver started at http://127.3.159.1:45317/ using document root <none> and password file <none>
I20260927 22:00:49.929978 3821 fs_manager.cc:359] Metadata directory not provided
I20260927 22:00:49.930111 3821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:00:49.930469 3821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:00:49.934299 3821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "4d964f2dd0d0407d9a8d6be50b08724a"
format_stamp: "Formatted at 2026-09-27 22:00:49 on dist-test-slave-sg4j"
server_key: "e06798119860a1c2d91bd9eb867ababe"
server_key_iv: "464e031fde2b01c43fc3ae7c02ec94cd"
server_key_version: "encryptionkey@0"
I20260927 22:00:49.936038 3821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "4d964f2dd0d0407d9a8d6be50b08724a"
format_stamp: "Formatted at 2026-09-27 22:00:49 on dist-test-slave-sg4j"
server_key: "e06798119860a1c2d91bd9eb867ababe"
server_key_iv: "464e031fde2b01c43fc3ae7c02ec94cd"
server_key_version: "encryptionkey@0"
I20260927 22:00:49.943904 3821 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260927 22:00:49.949195 3836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:49.951200 3821 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 22:00:49.951472 3821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "4d964f2dd0d0407d9a8d6be50b08724a"
format_stamp: "Formatted at 2026-09-27 22:00:49 on dist-test-slave-sg4j"
server_key: "e06798119860a1c2d91bd9eb867ababe"
server_key_iv: "464e031fde2b01c43fc3ae7c02ec94cd"
server_key_version: "encryptionkey@0"
I20260927 22:00:49.951762 3821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:00:49.980381 3821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:00:49.985845 3821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:00:49.986258 3821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:00:49.988246 3821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:00:49.990078 3821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:00:49.990195 3821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:49.990370 3821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:00:49.990451 3821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:50.040150 3821 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:43979
I20260927 22:00:50.040241 3948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:43979 every 8 connection(s)
I20260927 22:00:50.041945 3821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:00:50.049162 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 3821
I20260927 22:00:50.049603 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:00:50.050350 3708 external_mini_cluster.cc:1428] Setting key ca4db23bb24a8be8f331f3c1ac509094
Sep 27 22:00:50 dist-test-slave-sg4j krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546449, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:00:50.069141 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:50.045620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:35073 (local address 127.3.159.62:35275)
0927 22:00:50.046222 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:50.046227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:50.047891 (+ 1664us) server_negotiation.cc:407] Connection header received
0927 22:00:50.050749 (+ 2858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:50.050756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:50.050867 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:50.050997 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:50.053095 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:50.053920 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:50.055705 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:50.056141 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:50.056287 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:50.061316 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:50.061322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:50.061326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:50.061437 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:50.064861 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:50.066141 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:50.066149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:50.066154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:50.066336 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:50.066940 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:50.066950 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:50.066953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:50.067283 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:50.067473 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:50.068390 (+ 917us) server_negotiation.cc:299] Negotiation successful
0927 22:00:50.068692 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":294,"threads_started":1}
I20260927 22:00:50.070748 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:50.046512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:35275 (local address 127.3.159.1:35073)
0927 22:00:50.047635 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:50.047703 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:50.050326 (+ 2623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:50.051300 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:50.051311 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:50.052055 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:00:50.052889 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:50.052911 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:50.054105 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:50.054111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:50.055538 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:50.055551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:50.056394 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:50.056401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:50.056751 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:50.057570 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:00:50.057595 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:00:50.060907 (+ 3312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:00:50.065154 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:50.065168 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:50.065191 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:50.065954 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:50.066530 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:50.066537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:50.066547 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:50.066793 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:50.067436 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:00:50.067446 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:00:50.067943 (+ 497us) client_negotiation.cc:769] Sending connection context
0927 22:00:50.068272 (+ 329us) client_negotiation.cc:240] Negotiation successful
0927 22:00:50.068999 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":446,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:00:50.073915 3949 heartbeater.cc:347] Connected to a master server at 127.3.159.62:35275
I20260927 22:00:50.074368 3949 heartbeater.cc:464] Registering TS with master...
I20260927 22:00:50.075331 3949 heartbeater.cc:511] Master 127.3.159.62:35275 requested a full tablet report, sending...
I20260927 22:00:50.078850 3760 ts_manager.cc:194] Registered new tserver with Master: 4d964f2dd0d0407d9a8d6be50b08724a (127.3.159.1:43979)
I20260927 22:00:50.081205 3760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:35073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:00:50.101682 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35275
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:00:50.506454 3955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:00:50.506795 3955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:00:50.506970 3955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:00:50.515094 3955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:00:50.515233 3955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:00:50.515396 3955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:00:50.526351 3955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35275
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:00:50.527863 3955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:00:50.530011 3955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:00:50.546226 3963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:50.546458 3960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:50.548024 3961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:00:50.548453 3955 server_base.cc:1034] running on GCE node
I20260927 22:00:50.549467 3955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:00:50.550836 3955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:00:50.552150 3955 hybrid_clock.cc:648] HybridClock initialized: now 1790546450552085 us; error 68 us; skew 500 ppm
I20260927 22:00:50.552629 3955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:00:50 dist-test-slave-sg4j krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:00:50.557081 3955 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:00:50.559298 3955 webserver.cc:460] Webserver started at http://127.3.159.2:46789/ using document root <none> and password file <none>
I20260927 22:00:50.560293 3955 fs_manager.cc:359] Metadata directory not provided
I20260927 22:00:50.560428 3955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:00:50.560829 3955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:00:50.564742 3955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "59719f7e55dc473ab1f04c7f3a45d4ee"
format_stamp: "Formatted at 2026-09-27 22:00:50 on dist-test-slave-sg4j"
server_key: "48102c371464c98c614c70baac7bd473"
server_key_iv: "b2a92c3975aa2519051756385bb2392a"
server_key_version: "encryptionkey@0"
I20260927 22:00:50.565804 3955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "59719f7e55dc473ab1f04c7f3a45d4ee"
format_stamp: "Formatted at 2026-09-27 22:00:50 on dist-test-slave-sg4j"
server_key: "48102c371464c98c614c70baac7bd473"
server_key_iv: "b2a92c3975aa2519051756385bb2392a"
server_key_version: "encryptionkey@0"
I20260927 22:00:50.573149 3955 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 22:00:50.578588 3970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:50.581643 3955 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:00:50.581962 3955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "59719f7e55dc473ab1f04c7f3a45d4ee"
format_stamp: "Formatted at 2026-09-27 22:00:50 on dist-test-slave-sg4j"
server_key: "48102c371464c98c614c70baac7bd473"
server_key_iv: "b2a92c3975aa2519051756385bb2392a"
server_key_version: "encryptionkey@0"
I20260927 22:00:50.582243 3955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:00:50.603474 3955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:00:50.609478 3955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:00:50.609853 3955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:00:50.611126 3955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:00:50.612771 3955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:00:50.612861 3955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:50.612960 3955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:00:50.613009 3955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:50.661866 3955 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:46171
I20260927 22:00:50.662848 4082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:46171 every 8 connection(s)
I20260927 22:00:50.663995 3955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:00:50.667547 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 3955
I20260927 22:00:50.667781 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:00:50.668368 3708 external_mini_cluster.cc:1428] Setting key 623a061d3e4ee3a64b665a908651fe59
Sep 27 22:00:50 dist-test-slave-sg4j krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546450, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
I20260927 22:00:50.699219 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:50.667311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:37585 (local address 127.3.159.62:35275)
0927 22:00:50.667895 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:50.667904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:50.669966 (+ 2062us) server_negotiation.cc:407] Connection header received
0927 22:00:50.672289 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:50.672299 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:50.672416 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:50.672578 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:50.674973 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:50.676230 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:50.678178 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:50.679750 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:50.679952 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:50.690715 (+ 10763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:50.690723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:50.690728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:50.690784 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:50.695027 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:50.696189 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:50.696198 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:50.696204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:50.696408 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:50.696916 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:50.696923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:50.696929 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:50.697444 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:50.697619 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:50.698521 (+ 902us) server_negotiation.cc:299] Negotiation successful
0927 22:00:50.698840 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":341,"threads_started":1}
I20260927 22:00:50.700766 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:50.668251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:35275 (local address 127.3.159.2:37585)
0927 22:00:50.669350 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:50.669451 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:50.671878 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:50.672853 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:50.672864 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:50.673726 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:00:50.674624 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:50.674655 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:50.676458 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:50.676463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:50.677957 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:50.677987 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:50.680092 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:50.680099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:50.680468 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:50.686357 (+ 5889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:00:50.686400 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:00:50.690203 (+ 3803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:00:50.695336 (+ 5133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:50.695346 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:50.695363 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:50.695978 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:50.696543 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:50.696546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:50.696548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:50.696766 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:50.697598 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:00:50.697607 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:00:50.698012 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 22:00:50.698333 (+ 321us) client_negotiation.cc:240] Negotiation successful
0927 22:00:50.698891 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":406,"threads_started":1}
I20260927 22:00:50.702642 4083 heartbeater.cc:347] Connected to a master server at 127.3.159.62:35275
I20260927 22:00:50.702994 4083 heartbeater.cc:464] Registering TS with master...
I20260927 22:00:50.703824 4083 heartbeater.cc:511] Master 127.3.159.62:35275 requested a full tablet report, sending...
I20260927 22:00:50.705473 3760 ts_manager.cc:194] Registered new tserver with Master: 59719f7e55dc473ab1f04c7f3a45d4ee (127.3.159.2:46171)
I20260927 22:00:50.706410 3760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:37585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:00:50.720729 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35275
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:44507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:00:51.085985 3949 heartbeater.cc:503] Master 127.3.159.62:35275 was elected leader, sending a full tablet report...
W20260927 22:00:51.149655 4089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:00:51.150084 4089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:00:51.150269 4089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:00:51.158624 4089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:00:51.158826 4089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:00:51.159031 4089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:00:51.171690 4089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35275
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:00:51.173691 4089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:00:51.175514 4089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:00:51.193830 4097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:51.193904 4095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:51.193972 4094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:00:51.199571 4089 server_base.cc:1034] running on GCE node
I20260927 22:00:51.200547 4089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:00:51.201983 4089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:00:51.203336 4089 hybrid_clock.cc:648] HybridClock initialized: now 1790546451203257 us; error 62 us; skew 500 ppm
I20260927 22:00:51.203953 4089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:00:51 dist-test-slave-sg4j krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546451, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:00:51.208055 4089 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:00:51.210368 4089 webserver.cc:460] Webserver started at http://127.3.159.3:44131/ using document root <none> and password file <none>
I20260927 22:00:51.211277 4089 fs_manager.cc:359] Metadata directory not provided
I20260927 22:00:51.211397 4089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:00:51.211710 4089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:00:51.215528 4089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "e8938083d5cf419e872b56a160b35515"
format_stamp: "Formatted at 2026-09-27 22:00:51 on dist-test-slave-sg4j"
server_key: "4167c8bf18d805f32b7627930b02a3b6"
server_key_iv: "cffe209d2046a5a7e06700f5ceb1a784"
server_key_version: "encryptionkey@0"
I20260927 22:00:51.216387 4089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "e8938083d5cf419e872b56a160b35515"
format_stamp: "Formatted at 2026-09-27 22:00:51 on dist-test-slave-sg4j"
server_key: "4167c8bf18d805f32b7627930b02a3b6"
server_key_iv: "cffe209d2046a5a7e06700f5ceb1a784"
server_key_version: "encryptionkey@0"
I20260927 22:00:51.223764 4089 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260927 22:00:51.228974 4104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:51.231009 4089 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:00:51.231262 4089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "e8938083d5cf419e872b56a160b35515"
format_stamp: "Formatted at 2026-09-27 22:00:51 on dist-test-slave-sg4j"
server_key: "4167c8bf18d805f32b7627930b02a3b6"
server_key_iv: "cffe209d2046a5a7e06700f5ceb1a784"
server_key_version: "encryptionkey@0"
I20260927 22:00:51.231521 4089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:00:51.251652 4089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:00:51.257757 4089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:00:51.258136 4089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:00:51.259851 4089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:00:51.261715 4089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:00:51.261842 4089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:51.262022 4089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:00:51.262097 4089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:51.309751 4089 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:41679
I20260927 22:00:51.309906 4216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:41679 every 8 connection(s)
I20260927 22:00:51.311494 4089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:00:51.317158 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 4089
I20260927 22:00:51.317440 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:00:51.318022 3708 external_mini_cluster.cc:1428] Setting key 6b4de29532f22fd9015c0db92128899c
Sep 27 22:00:51 dist-test-slave-sg4j krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546451, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:00:51.339300 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:51.315326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:57169 (local address 127.3.159.62:35275)
0927 22:00:51.315887 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:51.315892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:51.317474 (+ 1582us) server_negotiation.cc:407] Connection header received
0927 22:00:51.320047 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:51.320054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:51.320188 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:51.320303 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:51.323653 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.324658 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.326739 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.327186 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.327343 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:51.331693 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:51.331698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:51.331702 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:51.331739 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:51.335026 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.336142 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.336149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.336153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.336418 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.337135 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.337139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.337142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.337651 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:51.337853 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:51.338683 (+ 830us) server_negotiation.cc:299] Negotiation successful
0927 22:00:51.338951 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":344,"threads_started":1}
I20260927 22:00:51.340550 4220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:51.316038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:35275 (local address 127.3.159.3:57169)
0927 22:00:51.317117 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:51.317224 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:51.319652 (+ 2428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:51.321118 (+ 1466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:51.321135 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:51.322639 (+ 1504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:00:51.323416 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:51.323438 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.324887 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.324892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:51.326535 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:51.326552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.327476 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.327484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:51.327749 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:51.328563 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:00:51.328589 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:00:51.331357 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:00:51.335228 (+ 3871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:51.335241 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:51.335270 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:51.335903 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:51.336627 (+ 724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:51.336633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:51.336639 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:51.336960 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:51.337822 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:00:51.337831 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:00:51.338276 (+ 445us) client_negotiation.cc:769] Sending connection context
0927 22:00:51.338548 (+ 272us) client_negotiation.cc:240] Negotiation successful
0927 22:00:51.339059 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":362,"threads_started":1}
I20260927 22:00:51.342705 4217 heartbeater.cc:347] Connected to a master server at 127.3.159.62:35275
I20260927 22:00:51.343142 4217 heartbeater.cc:464] Registering TS with master...
I20260927 22:00:51.344143 4217 heartbeater.cc:511] Master 127.3.159.62:35275 requested a full tablet report, sending...
I20260927 22:00:51.346632 3760 ts_manager.cc:194] Registered new tserver with Master: e8938083d5cf419e872b56a160b35515 (127.3.159.3:41679)
I20260927 22:00:51.347728 3760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:57169
I20260927 22:00:51.351082 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:00:51 dist-test-slave-sg4j krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:00:51 dist-test-slave-sg4j krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:00:51.410681 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:51.390392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.3.159.1:43979)
0927 22:00:51.391371 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:51.391379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:51.391468 (+ 89us) server_negotiation.cc:407] Connection header received
0927 22:00:51.391542 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:51.391553 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:51.391880 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:51.392042 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:51.394139 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.395220 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.396788 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.397655 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.397791 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:51.402980 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:51.402997 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:51.403017 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:51.403068 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:51.406930 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.407914 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.407942 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.407945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.408185 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.408743 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.408747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.408749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.409226 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:51.409617 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:51.409983 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 22:00:51.410258 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":538,"threads_started":1}
W20260927 22:00:51.412820 3864 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45168
Sep 27 22:00:51 dist-test-slave-sg4j krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:00:51 dist-test-slave-sg4j krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:00:51.468590 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:51.450669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.3.159.1:43979)
0927 22:00:51.450937 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:51.450951 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:51.451154 (+ 203us) server_negotiation.cc:407] Connection header received
0927 22:00:51.451423 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:51.451428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:51.451542 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:51.451676 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:51.453354 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.454234 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.455799 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.456232 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.456446 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:51.461528 (+ 5082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:51.461535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:51.461539 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:51.461680 (+ 141us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:51.464973 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.466224 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.466229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.466232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.466416 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.466956 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.466960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.466962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.467281 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:51.467492 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:51.468028 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 22:00:51.468246 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 22:00:51.471043 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 3821
I20260927 22:00:51.488042 4225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:51.472376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.3.159.1:43979)
0927 22:00:51.472705 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:51.472711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:51.472997 (+ 286us) server_negotiation.cc:407] Connection header received
0927 22:00:51.473139 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:51.473145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:51.473209 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:51.473321 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:51.474884 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.475621 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.478214 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.478763 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.478911 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:51.481713 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:51.481718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:51.481722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:51.481770 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:51.484830 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.485830 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.485835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.485838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.486021 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.486515 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.486519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.486521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.486855 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:51.487050 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:51.487470 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 22:00:51.487648 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260927 22:00:51.488953 3944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:00:51.580044 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 3821
I20260927 22:00:51.601788 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 3955
Sep 27 22:00:51 dist-test-slave-sg4j krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:00:51.621625 4232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:51.603036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.3.159.2:46171)
0927 22:00:51.603700 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:51.603707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:51.603738 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:00:51.603802 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:51.603809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:51.604034 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:51.604180 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:51.605762 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.606798 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.608284 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.609025 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.609153 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:51.614282 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:51.614303 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:51.614324 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:51.614388 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:51.618020 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.618990 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.619000 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.619003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.619188 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.619698 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.619702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.619704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.620213 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:51.620488 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:51.620892 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 22:00:51.621149 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":371,"threads_started":1}
I20260927 22:00:51.623273 4078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:00:51.715754 4083 heartbeater.cc:503] Master 127.3.159.62:35275 was elected leader, sending a full tablet report...
I20260927 22:00:51.717139 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 3955
W20260927 22:00:51.741074 3744 connection.cc:535] server connection from 127.3.159.2:37585 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 22:00:51.741818 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 4089
Sep 27 22:00:51 dist-test-slave-sg4j krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:00:51.763168 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:51.743312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.3.159.3:41679)
0927 22:00:51.744094 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:51.744104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:51.744148 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:00:51.744229 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:51.744237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:51.744473 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:51.744669 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:51.746815 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.747965 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.749872 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:51.750625 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:51.750931 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:51.756299 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:51.756316 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:51.756336 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:51.756381 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:51.759670 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.760709 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.760719 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.760722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.760888 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:51.761355 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:51.761360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:51.761362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:51.761918 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:51.762182 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:51.762636 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 22:00:51.762864 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":382,"threads_started":1}
I20260927 22:00:51.765142 4212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:00:51.860800 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 4089
I20260927 22:00:51.878305 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 3728
I20260927 22:00:51.879920 3790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:00:51.987638 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 3728
2026-09-27T22:00:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4697 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:00:52 dist-test-slave-sg4j krb5kdc[4239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:00:52 dist-test-slave-sg4j krb5kdc[4239](info): set up 2 sockets
Sep 27 22:00:52 dist-test-slave-sg4j krb5kdc[4239](info): commencing operation
krb5kdc: starting...
W20260927 22:00:53.106034 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:00:53 dist-test-slave-sg4j krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:00:53.322832 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:35273
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:35273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:00:53.740692 4252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:00:53.741044 4252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:00:53.741145 4252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:00:53.748835 4252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:00:53.748967 4252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:00:53.749017 4252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:00:53.749068 4252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:00:53.749119 4252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:00:53.759930 4252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:35273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:35273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:00:53.761650 4252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:00:53.763499 4252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:00:53.778913 4257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:53.778996 4258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:53.778828 4260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:00:53.780881 4252 server_base.cc:1034] running on GCE node
I20260927 22:00:53.781966 4252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:00:53.783742 4252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:00:53.785056 4252 hybrid_clock.cc:648] HybridClock initialized: now 1790546453784977 us; error 65 us; skew 500 ppm
I20260927 22:00:53.785655 4252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:00:53 dist-test-slave-sg4j krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546453, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:00:53.789530 4252 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:00:53.791460 4252 webserver.cc:460] Webserver started at http://127.3.159.62:33615/ using document root <none> and password file <none>
I20260927 22:00:53.792312 4252 fs_manager.cc:359] Metadata directory not provided
I20260927 22:00:53.792471 4252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:00:53.792819 4252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:00:53.796159 4252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "3ace0c863af144cbb6b88bb02fb78bc0"
format_stamp: "Formatted at 2026-09-27 22:00:53 on dist-test-slave-sg4j"
server_key: "650a18a690095a8bbe6cf4040f2db822"
server_key_iv: "ec5f9b7cea2ed3cd7d5cbad57d015d7f"
server_key_version: "encryptionkey@0"
I20260927 22:00:53.797012 4252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "3ace0c863af144cbb6b88bb02fb78bc0"
format_stamp: "Formatted at 2026-09-27 22:00:53 on dist-test-slave-sg4j"
server_key: "650a18a690095a8bbe6cf4040f2db822"
server_key_iv: "ec5f9b7cea2ed3cd7d5cbad57d015d7f"
server_key_version: "encryptionkey@0"
I20260927 22:00:53.803547 4252 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 22:00:53.808357 4267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:53.810403 4252 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 22:00:53.810693 4252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "3ace0c863af144cbb6b88bb02fb78bc0"
format_stamp: "Formatted at 2026-09-27 22:00:53 on dist-test-slave-sg4j"
server_key: "650a18a690095a8bbe6cf4040f2db822"
server_key_iv: "ec5f9b7cea2ed3cd7d5cbad57d015d7f"
server_key_version: "encryptionkey@0"
I20260927 22:00:53.810933 4252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:00:53.831334 4252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:00:53.836592 4252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:00:53.836946 4252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:00:53.861336 4252 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:35273
I20260927 22:00:53.861523 4318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:35273 every 8 connection(s)
I20260927 22:00:53.863085 4252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:00:53.868659 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 4252
I20260927 22:00:53.868746 4319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:00:53.868991 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:00:53.869926 3708 external_mini_cluster.cc:1428] Setting key 4f20328cba2370a19446de2e25079208
I20260927 22:00:53.886554 4319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0: Bootstrap starting.
Sep 27 22:00:53 dist-test-slave-sg4j krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:00:53.891036 4319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0: Neither blocks nor log segments found. Creating new log.
I20260927 22:00:53.892477 4319 log.cc:824] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0: Log is configured to *not* fsync() on all Append() calls
I20260927 22:00:53.897099 4319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0: No bootstrap required, opened a new log
I20260927 22:00:53.901904 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:53.872994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.3.159.62:35273)
0927 22:00:53.874128 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:53.874158 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:53.874255 (+ 97us) server_negotiation.cc:407] Connection header received
0927 22:00:53.876918 (+ 2663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:53.876968 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:53.877744 (+ 776us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:53.878284 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:53.880405 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:53.882287 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:53.884401 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:53.885524 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:53.885752 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:53.891367 (+ 5615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:53.891382 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:53.891410 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:53.891480 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:53.895766 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:53.896725 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:53.896733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:53.896741 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:53.896949 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:53.897498 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:53.897507 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:53.897511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:53.898658 (+ 1147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:53.899050 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:53.899552 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 22:00:53.900178 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":542,"threads_started":1}
I20260927 22:00:53.908067 4319 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ace0c863af144cbb6b88bb02fb78bc0" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35273 } }
I20260927 22:00:53.908497 4319 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:00:53.908623 4319 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ace0c863af144cbb6b88bb02fb78bc0, State: Initialized, Role: FOLLOWER
I20260927 22:00:53.909513 4319 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ace0c863af144cbb6b88bb02fb78bc0" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35273 } }
I20260927 22:00:53.909999 4319 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:00:53.910264 4319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:00:53.910560 4319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:00:53.913106 4319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ace0c863af144cbb6b88bb02fb78bc0" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35273 } }
I20260927 22:00:53.913940 4319 leader_election.cc:302] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ace0c863af144cbb6b88bb02fb78bc0; no voters:
I20260927 22:00:53.914671 4319 leader_election.cc:288] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:00:53.914922 4324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:00:53.915850 4324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [term 1 LEADER]: Becoming Leader. State: Replica: 3ace0c863af144cbb6b88bb02fb78bc0, State: Running, Role: LEADER
I20260927 22:00:53.916643 4324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ace0c863af144cbb6b88bb02fb78bc0" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35273 } }
I20260927 22:00:53.917496 4319 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:00:53.921056 4326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ace0c863af144cbb6b88bb02fb78bc0. Latest consensus state: current_term: 1 leader_uuid: "3ace0c863af144cbb6b88bb02fb78bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ace0c863af144cbb6b88bb02fb78bc0" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35273 } } }
I20260927 22:00:53.921365 4325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ace0c863af144cbb6b88bb02fb78bc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ace0c863af144cbb6b88bb02fb78bc0" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35273 } } }
I20260927 22:00:53.921887 4326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:00:53.921880 4325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0 [sys.catalog]: This master's current role is: LEADER
I20260927 22:00:53.925652 4333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:00:53.931186 4333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:00:53.945174 4333 catalog_manager.cc:1347] Generated new cluster ID: 08671c7cd23340cab15063171995c21c
I20260927 22:00:53.945536 4333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:00:53.957718 4333 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:00:53.960244 4333 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:00:53.974468 4333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ace0c863af144cbb6b88bb02fb78bc0: Generated new TSK 0
I20260927 22:00:53.975703 4333 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:00:54.030833 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35273
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:00:54.442509 4345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:00:54.442834 4345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:00:54.442978 4345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:00:54.452139 4345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:00:54.452320 4345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:00:54.452525 4345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:00:54.466868 4345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:00:54.468647 4345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:00:54.470314 4345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:00:54.489071 4353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:54.489106 4351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:54.489179 4350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:00:54.490460 4345 server_base.cc:1034] running on GCE node
I20260927 22:00:54.491302 4345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:00:54.492779 4345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:00:54.494058 4345 hybrid_clock.cc:648] HybridClock initialized: now 1790546454494008 us; error 56 us; skew 500 ppm
I20260927 22:00:54.494545 4345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:00:54 dist-test-slave-sg4j krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:00:54.498660 4345 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:00:54.500700 4345 webserver.cc:460] Webserver started at http://127.3.159.1:46689/ using document root <none> and password file <none>
I20260927 22:00:54.501698 4345 fs_manager.cc:359] Metadata directory not provided
I20260927 22:00:54.501863 4345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:00:54.502239 4345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:00:54.506245 4345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "5ab2be8055314512980a4eee5e3d9641"
format_stamp: "Formatted at 2026-09-27 22:00:54 on dist-test-slave-sg4j"
server_key: "a839cac85d58fd9b8e751414420e00b3"
server_key_iv: "02c691e0f24b767bd21654054fdb08dd"
server_key_version: "encryptionkey@0"
I20260927 22:00:54.507452 4345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "5ab2be8055314512980a4eee5e3d9641"
format_stamp: "Formatted at 2026-09-27 22:00:54 on dist-test-slave-sg4j"
server_key: "a839cac85d58fd9b8e751414420e00b3"
server_key_iv: "02c691e0f24b767bd21654054fdb08dd"
server_key_version: "encryptionkey@0"
I20260927 22:00:54.514839 4345 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 22:00:54.520169 4360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:54.522364 4345 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 22:00:54.522634 4345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "5ab2be8055314512980a4eee5e3d9641"
format_stamp: "Formatted at 2026-09-27 22:00:54 on dist-test-slave-sg4j"
server_key: "a839cac85d58fd9b8e751414420e00b3"
server_key_iv: "02c691e0f24b767bd21654054fdb08dd"
server_key_version: "encryptionkey@0"
I20260927 22:00:54.522912 4345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:00:54.544713 4345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:00:54.550479 4345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:00:54.550845 4345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:00:54.552124 4345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:00:54.554013 4345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:00:54.554133 4345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:54.554306 4345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:00:54.554384 4345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:54.596688 4345 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:43433
I20260927 22:00:54.596913 4472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:43433 every 8 connection(s)
I20260927 22:00:54.598449 4345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:00:54.605782 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 4345
I20260927 22:00:54.606029 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:00:54.606747 3708 external_mini_cluster.cc:1428] Setting key 8213e0e27772d7b1a45f3e3e68242a99
Sep 27 22:00:54 dist-test-slave-sg4j krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:00:54.629330 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:54.601842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:52653 (local address 127.3.159.62:35273)
0927 22:00:54.602356 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:54.602360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:54.603678 (+ 1318us) server_negotiation.cc:407] Connection header received
0927 22:00:54.606205 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:54.606212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:54.606327 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:54.606473 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:54.609436 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:54.610341 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:54.613557 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:54.614030 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:54.614166 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:54.619544 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:54.619557 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:54.619565 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:54.619636 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:54.624130 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:54.625833 (+ 1703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:54.625843 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:54.625850 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:54.626034 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:54.626876 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:54.626885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:54.626890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:54.627358 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:54.627545 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:54.628644 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0927 22:00:54.629003 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":294,"threads_started":1}
I20260927 22:00:54.630479 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:54.602456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:35273 (local address 127.3.159.1:52653)
0927 22:00:54.603432 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:54.603486 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:54.605616 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:54.606866 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:54.606879 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:54.608160 (+ 1281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:00:54.609152 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:54.609188 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:54.610613 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:54.610622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:54.613242 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:54.613271 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:54.614385 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:54.614397 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:54.614820 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:54.615864 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:00:54.615902 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:00:54.619127 (+ 3225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:00:54.624452 (+ 5325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:54.624465 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:54.624488 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:54.625635 (+ 1147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:54.626294 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:54.626300 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:54.626303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:54.626659 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:54.627583 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:00:54.627591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:00:54.628121 (+ 530us) client_negotiation.cc:769] Sending connection context
0927 22:00:54.628440 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 22:00:54.629049 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":365,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:00:54.633165 4473 heartbeater.cc:347] Connected to a master server at 127.3.159.62:35273
I20260927 22:00:54.633694 4473 heartbeater.cc:464] Registering TS with master...
I20260927 22:00:54.634768 4473 heartbeater.cc:511] Master 127.3.159.62:35273 requested a full tablet report, sending...
I20260927 22:00:54.638335 4284 ts_manager.cc:194] Registered new tserver with Master: 5ab2be8055314512980a4eee5e3d9641 (127.3.159.1:43433)
I20260927 22:00:54.641323 4284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:52653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:00:54.663254 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35273
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:00:55.082566 4479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:00:55.082974 4479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:00:55.083083 4479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:00:55.091827 4479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:00:55.092010 4479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:00:55.092154 4479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:00:55.104148 4479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:00:55.106005 4479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:00:55.107805 4479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:00:55.126317 4484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:55.126366 4485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:55.126219 4487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:00:55.127604 4479 server_base.cc:1034] running on GCE node
I20260927 22:00:55.128566 4479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:00:55.130151 4479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:00:55.131534 4479 hybrid_clock.cc:648] HybridClock initialized: now 1790546455131443 us; error 90 us; skew 500 ppm
I20260927 22:00:55.132005 4479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:00:55 dist-test-slave-sg4j krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:00:55.137151 4479 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:00:55.139472 4479 webserver.cc:460] Webserver started at http://127.3.159.2:40753/ using document root <none> and password file <none>
I20260927 22:00:55.140581 4479 fs_manager.cc:359] Metadata directory not provided
I20260927 22:00:55.140782 4479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:00:55.141217 4479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:00:55.145686 4479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "cc918b6fdf6f4f378b5163331ab1ca62"
format_stamp: "Formatted at 2026-09-27 22:00:55 on dist-test-slave-sg4j"
server_key: "5f1aa05ab5be2531989b14f49f8004c8"
server_key_iv: "0a1a90fde55aad019141363117442f5f"
server_key_version: "encryptionkey@0"
I20260927 22:00:55.146844 4479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "cc918b6fdf6f4f378b5163331ab1ca62"
format_stamp: "Formatted at 2026-09-27 22:00:55 on dist-test-slave-sg4j"
server_key: "5f1aa05ab5be2531989b14f49f8004c8"
server_key_iv: "0a1a90fde55aad019141363117442f5f"
server_key_version: "encryptionkey@0"
I20260927 22:00:55.154340 4479 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:00:55.159724 4494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:55.162475 4479 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:00:55.162818 4479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "cc918b6fdf6f4f378b5163331ab1ca62"
format_stamp: "Formatted at 2026-09-27 22:00:55 on dist-test-slave-sg4j"
server_key: "5f1aa05ab5be2531989b14f49f8004c8"
server_key_iv: "0a1a90fde55aad019141363117442f5f"
server_key_version: "encryptionkey@0"
I20260927 22:00:55.163152 4479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:00:55.183022 4479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:00:55.188825 4479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:00:55.189240 4479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:00:55.190644 4479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:00:55.192428 4479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:00:55.192545 4479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:55.192713 4479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:00:55.192794 4479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:55.237010 4479 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:43061
I20260927 22:00:55.237075 4606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:43061 every 8 connection(s)
I20260927 22:00:55.238785 4479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:00:55.241094 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 4479
I20260927 22:00:55.241314 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:00:55.241851 3708 external_mini_cluster.cc:1428] Setting key 75308a709f940f1bb2b13edeb5aa2ee2
Sep 27 22:00:55 dist-test-slave-sg4j krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
I20260927 22:00:55.268553 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:55.242573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:44529 (local address 127.3.159.62:35273)
0927 22:00:55.243434 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:55.243440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:55.244909 (+ 1469us) server_negotiation.cc:407] Connection header received
0927 22:00:55.247525 (+ 2616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:55.247532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:55.247656 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:55.248012 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:55.251041 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:55.251888 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:55.253886 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:55.254343 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:55.254503 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:55.259769 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:55.259778 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:55.259782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:55.259833 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:55.263864 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:55.265057 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:55.265063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:55.265066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:55.265264 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:55.265941 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:55.265958 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:55.265961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:55.266483 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:55.266690 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:55.267748 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0927 22:00:55.268078 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":303,"threads_started":1}
I20260927 22:00:55.270514 4610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:55.243429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:35273 (local address 127.3.159.2:44529)
0927 22:00:55.244693 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:55.244756 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:55.247060 (+ 2304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:55.248376 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:55.248389 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:55.249055 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:00:55.250524 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:55.250562 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:55.252137 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:55.252142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:55.253679 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:55.253699 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:55.254629 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:55.254636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:55.254894 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:55.256046 (+ 1152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:00:55.256074 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:00:55.259223 (+ 3149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:00:55.264166 (+ 4943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:55.264178 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:55.264201 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:55.264851 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:55.265437 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:55.265447 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:55.265450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:55.265714 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:55.266674 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:00:55.266687 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:00:55.267229 (+ 542us) client_negotiation.cc:769] Sending connection context
0927 22:00:55.267613 (+ 384us) client_negotiation.cc:240] Negotiation successful
0927 22:00:55.268387 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":421,"threads_started":1}
I20260927 22:00:55.272893 4607 heartbeater.cc:347] Connected to a master server at 127.3.159.62:35273
I20260927 22:00:55.273568 4607 heartbeater.cc:464] Registering TS with master...
I20260927 22:00:55.274734 4607 heartbeater.cc:511] Master 127.3.159.62:35273 requested a full tablet report, sending...
I20260927 22:00:55.277364 4284 ts_manager.cc:194] Registered new tserver with Master: cc918b6fdf6f4f378b5163331ab1ca62 (127.3.159.2:43061)
I20260927 22:00:55.279040 4284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:44529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:00:55.303936 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35273
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:00:55.646034 4473 heartbeater.cc:503] Master 127.3.159.62:35273 was elected leader, sending a full tablet report...
W20260927 22:00:55.754503 4613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:00:55.755087 4613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:00:55.755303 4613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:00:55.768129 4613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:00:55.768373 4613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:00:55.768615 4613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:00:55.786484 4613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:35273
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:00:55.788440 4613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:00:55.790297 4613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:00:55.809697 4618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:55.809675 4619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:55.809697 4621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:00:55.811581 4613 server_base.cc:1034] running on GCE node
I20260927 22:00:55.812594 4613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:00:55.814291 4613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:00:55.815619 4613 hybrid_clock.cc:648] HybridClock initialized: now 1790546455815539 us; error 86 us; skew 500 ppm
I20260927 22:00:55.816186 4613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:00:55 dist-test-slave-sg4j krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:00:55.820830 4613 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:00:55.822815 4613 webserver.cc:460] Webserver started at http://127.3.159.3:45865/ using document root <none> and password file <none>
I20260927 22:00:55.824007 4613 fs_manager.cc:359] Metadata directory not provided
I20260927 22:00:55.824149 4613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:00:55.824517 4613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:00:55.828645 4613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "1385ac32bdeb44f7ae193ea43980a088"
format_stamp: "Formatted at 2026-09-27 22:00:55 on dist-test-slave-sg4j"
server_key: "b4cf64e43175b30e333e0bc88e4cf2f6"
server_key_iv: "f3fb20c1fedb51a28c99fc57b7d8158b"
server_key_version: "encryptionkey@0"
I20260927 22:00:55.829821 4613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "1385ac32bdeb44f7ae193ea43980a088"
format_stamp: "Formatted at 2026-09-27 22:00:55 on dist-test-slave-sg4j"
server_key: "b4cf64e43175b30e333e0bc88e4cf2f6"
server_key_iv: "f3fb20c1fedb51a28c99fc57b7d8158b"
server_key_version: "encryptionkey@0"
I20260927 22:00:55.837284 4613 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 22:00:55.842135 4628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:55.844630 4613 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:00:55.844944 4613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "1385ac32bdeb44f7ae193ea43980a088"
format_stamp: "Formatted at 2026-09-27 22:00:55 on dist-test-slave-sg4j"
server_key: "b4cf64e43175b30e333e0bc88e4cf2f6"
server_key_iv: "f3fb20c1fedb51a28c99fc57b7d8158b"
server_key_version: "encryptionkey@0"
I20260927 22:00:55.845257 4613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:00:55.867225 4613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:00:55.874396 4613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:00:55.874713 4613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:00:55.875882 4613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:00:55.877550 4613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:00:55.877655 4613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:55.877748 4613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:00:55.877795 4613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:55.924055 4613 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:36081
I20260927 22:00:55.924388 4740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:36081 every 8 connection(s)
I20260927 22:00:55.926033 4613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:00:55.931317 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 4613
I20260927 22:00:55.931545 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:00:55.932031 3708 external_mini_cluster.cc:1428] Setting key 9ee54ece1b5f9924191421e2a466d8dc
Sep 27 22:00:55 dist-test-slave-sg4j krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:00:55.955679 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:55.929861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:52817 (local address 127.3.159.62:35273)
0927 22:00:55.930362 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:55.930367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:55.932193 (+ 1826us) server_negotiation.cc:407] Connection header received
0927 22:00:55.935261 (+ 3068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:55.935267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:55.935380 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:55.935537 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:55.937900 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:55.938851 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:55.941201 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:55.941832 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:55.941994 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:55.947026 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:55.947033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:55.947037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:55.947092 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:55.951440 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:55.952610 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:55.952616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:55.952619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:55.952775 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:55.953337 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:55.953345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:55.953354 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:55.953762 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:55.953954 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:55.954902 (+ 948us) server_negotiation.cc:299] Negotiation successful
0927 22:00:55.955204 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
I20260927 22:00:55.957124 4744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:55.930616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:35273 (local address 127.3.159.3:52817)
0927 22:00:55.931869 (+ 1253us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:55.931945 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:55.934830 (+ 2885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:55.935908 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:55.935922 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:55.936681 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:00:55.937629 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:55.937657 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:55.939060 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:55.939065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:55.940987 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:55.941007 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:55.942132 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:55.942140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:55.942441 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:55.943410 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:00:55.943445 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:00:55.946621 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:00:55.951687 (+ 5066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:55.951702 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:55.951739 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:55.952424 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:55.952926 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:55.952932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:55.952936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:55.953195 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:55.953916 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:00:55.953929 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:00:55.954436 (+ 507us) client_negotiation.cc:769] Sending connection context
0927 22:00:55.954774 (+ 338us) client_negotiation.cc:240] Negotiation successful
0927 22:00:55.955523 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":513,"threads_started":1}
I20260927 22:00:55.959584 4741 heartbeater.cc:347] Connected to a master server at 127.3.159.62:35273
I20260927 22:00:55.960202 4741 heartbeater.cc:464] Registering TS with master...
I20260927 22:00:55.961457 4741 heartbeater.cc:511] Master 127.3.159.62:35273 requested a full tablet report, sending...
I20260927 22:00:55.963326 4284 ts_manager.cc:194] Registered new tserver with Master: 1385ac32bdeb44f7ae193ea43980a088 (127.3.159.3:36081)
I20260927 22:00:55.964265 4284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:52817
I20260927 22:00:55.976410 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:00:56.002357 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:55.985648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.3.159.62:35273)
0927 22:00:55.985892 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:55.985898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:55.986552 (+ 654us) server_negotiation.cc:407] Connection header received
0927 22:00:55.986743 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:55.986748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:55.986836 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:55.986968 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:55.988945 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:55.990516 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:55.992290 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:55.992884 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:55.993068 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:55.996022 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:55.996029 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:55.996033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:55.996099 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:55.999123 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.000004 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.000008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.000011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.000180 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.000625 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.000631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.000634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.001028 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:56.001267 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.001897 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.002101 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 22:00:56.011549 4284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:00:56.020998 4284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:00:56.065096 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.042512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.3.159.2:43061)
0927 22:00:56.055330 (+ 12818us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:56.055345 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:56.056085 (+ 740us) server_negotiation.cc:407] Connection header received
0927 22:00:56.056246 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:56.056258 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:56.056611 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:56.056882 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:00:56.057565 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.059523 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.062305 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.063348 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.063532 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.063770 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.064532 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.064728 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12530,"thread_start_us":12311,"threads_started":1}
I20260927 22:00:56.065308 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.041906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:43061 (local address 127.0.0.1:55762)
0927 22:00:56.055737 (+ 13831us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:56.055764 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:56.055939 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:56.056909 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:56.056917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:56.056958 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:00:56.057332 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:56.057342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.059692 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.059697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:56.062115 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:56.062128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.063888 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.063896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:56.064152 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.064239 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 22:00:56.064405 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 22:00:56.064659 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13673,"mutex_wait_us":29,"thread_start_us":8966,"threads_started":1}
I20260927 22:00:56.067888 4754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.039910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:43433 (local address 127.0.0.1:35470)
0927 22:00:56.041535 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:56.048401 (+ 6866us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:56.048808 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:56.057630 (+ 8822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:56.057635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:56.057679 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:00:56.059073 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:56.059092 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.061044 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.061052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:56.063904 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:56.063921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.066728 (+ 2807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.066740 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:56.066978 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.067024 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:00:56.067172 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 22:00:56.067374 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"mutex_wait_us":30,"spinlock_wait_cycles":2944,"thread_start_us":1072,"threads_started":1}
I20260927 22:00:56.068411 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.040258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.3.159.1:43433)
0927 22:00:56.056765 (+ 16507us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:56.056777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:56.056817 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:00:56.056949 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:56.056959 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:56.057302 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:56.058050 (+ 748us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:00:56.059379 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.060877 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.064150 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.066478 (+ 2328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.066627 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.066883 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.067410 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.067673 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16260,"thread_start_us":16016,"threads_started":1}
I20260927 22:00:56.069480 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.040642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:36081 (local address 127.0.0.1:38772)
0927 22:00:56.051348 (+ 10706us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:56.051382 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:56.051574 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:56.057622 (+ 6048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:56.057630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:56.057668 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:00:56.059080 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:56.059097 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.061410 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.061416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:56.065721 (+ 4305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:56.065745 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.068305 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.068330 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:56.068632 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.068682 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 22:00:56.068814 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 22:00:56.068983 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10455,"mutex_wait_us":1883,"thread_start_us":15278,"threads_started":1}
I20260927 22:00:56.069796 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.041658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.3.159.3:36081)
0927 22:00:56.056917 (+ 15259us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:56.056926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:56.056958 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:00:56.057051 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:56.057058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:56.057302 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:56.057783 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:00:56.059439 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.061186 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.065969 (+ 4783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.068025 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.068233 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.068477 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.068907 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.069161 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14940,"spinlock_wait_cycles":37248,"thread_start_us":14888,"threads_started":1}
I20260927 22:00:56.070981 4542 tablet_service.cc:1467] Processing CreateTablet for tablet 30649c6695914b87bb3af59d7f7adf6c (DEFAULT_TABLE table=test-table [id=099cfae4b2af4483a7efd2c75329e48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:00:56.073742 4542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30649c6695914b87bb3af59d7f7adf6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:00:56.073884 4408 tablet_service.cc:1467] Processing CreateTablet for tablet 30649c6695914b87bb3af59d7f7adf6c (DEFAULT_TABLE table=test-table [id=099cfae4b2af4483a7efd2c75329e48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:00:56.076249 4676 tablet_service.cc:1467] Processing CreateTablet for tablet 30649c6695914b87bb3af59d7f7adf6c (DEFAULT_TABLE table=test-table [id=099cfae4b2af4483a7efd2c75329e48e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:00:56.077152 4408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30649c6695914b87bb3af59d7f7adf6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:00:56.078770 4676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30649c6695914b87bb3af59d7f7adf6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:00:56.094036 4760 tablet_bootstrap.cc:492] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62: Bootstrap starting.
I20260927 22:00:56.098016 4761 tablet_bootstrap.cc:492] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641: Bootstrap starting.
I20260927 22:00:56.099542 4762 tablet_bootstrap.cc:492] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088: Bootstrap starting.
I20260927 22:00:56.101366 4760 tablet_bootstrap.cc:654] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62: Neither blocks nor log segments found. Creating new log.
I20260927 22:00:56.103044 4761 tablet_bootstrap.cc:654] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641: Neither blocks nor log segments found. Creating new log.
I20260927 22:00:56.103395 4760 log.cc:824] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62: Log is configured to *not* fsync() on all Append() calls
I20260927 22:00:56.104491 4762 tablet_bootstrap.cc:654] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088: Neither blocks nor log segments found. Creating new log.
I20260927 22:00:56.104583 4761 log.cc:824] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641: Log is configured to *not* fsync() on all Append() calls
I20260927 22:00:56.107015 4762 log.cc:824] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088: Log is configured to *not* fsync() on all Append() calls
I20260927 22:00:56.108522 4760 tablet_bootstrap.cc:492] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62: No bootstrap required, opened a new log
I20260927 22:00:56.108886 4761 tablet_bootstrap.cc:492] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641: No bootstrap required, opened a new log
I20260927 22:00:56.108968 4760 ts_tablet_manager.cc:1397] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260927 22:00:56.109295 4761 ts_tablet_manager.cc:1397] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260927 22:00:56.111665 4762 tablet_bootstrap.cc:492] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088: No bootstrap required, opened a new log
I20260927 22:00:56.112071 4762 ts_tablet_manager.cc:1397] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.009s
I20260927 22:00:56.119122 4761 raft_consensus.cc:348] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab2be8055314512980a4eee5e3d9641" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43433 } } peers { permanent_uuid: "1385ac32bdeb44f7ae193ea43980a088" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36081 } } peers { permanent_uuid: "cc918b6fdf6f4f378b5163331ab1ca62" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43061 } }
I20260927 22:00:56.119583 4761 raft_consensus.cc:374] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:00:56.119674 4761 raft_consensus.cc:729] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ab2be8055314512980a4eee5e3d9641, State: Initialized, Role: FOLLOWER
I20260927 22:00:56.120432 4761 consensus_queue.cc:260] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab2be8055314512980a4eee5e3d9641" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43433 } } peers { permanent_uuid: "1385ac32bdeb44f7ae193ea43980a088" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36081 } } peers { permanent_uuid: "cc918b6fdf6f4f378b5163331ab1ca62" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43061 } }
I20260927 22:00:56.122253 4760 raft_consensus.cc:348] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab2be8055314512980a4eee5e3d9641" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43433 } } peers { permanent_uuid: "1385ac32bdeb44f7ae193ea43980a088" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36081 } } peers { permanent_uuid: "cc918b6fdf6f4f378b5163331ab1ca62" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43061 } }
I20260927 22:00:56.122942 4760 raft_consensus.cc:374] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:00:56.123085 4760 raft_consensus.cc:729] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc918b6fdf6f4f378b5163331ab1ca62, State: Initialized, Role: FOLLOWER
I20260927 22:00:56.124046 4760 consensus_queue.cc:260] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab2be8055314512980a4eee5e3d9641" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43433 } } peers { permanent_uuid: "1385ac32bdeb44f7ae193ea43980a088" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36081 } } peers { permanent_uuid: "cc918b6fdf6f4f378b5163331ab1ca62" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43061 } }
I20260927 22:00:56.125622 4762 raft_consensus.cc:348] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab2be8055314512980a4eee5e3d9641" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43433 } } peers { permanent_uuid: "1385ac32bdeb44f7ae193ea43980a088" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36081 } } peers { permanent_uuid: "cc918b6fdf6f4f378b5163331ab1ca62" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43061 } }
I20260927 22:00:56.127215 4762 raft_consensus.cc:374] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:00:56.127352 4762 raft_consensus.cc:729] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1385ac32bdeb44f7ae193ea43980a088, State: Initialized, Role: FOLLOWER
I20260927 22:00:56.128015 4607 heartbeater.cc:503] Master 127.3.159.62:35273 was elected leader, sending a full tablet report...
I20260927 22:00:56.128367 4762 consensus_queue.cc:260] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab2be8055314512980a4eee5e3d9641" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43433 } } peers { permanent_uuid: "1385ac32bdeb44f7ae193ea43980a088" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36081 } } peers { permanent_uuid: "cc918b6fdf6f4f378b5163331ab1ca62" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43061 } }
I20260927 22:00:56.128751 4760 ts_tablet_manager.cc:1428] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62: Time spent starting tablet: real 0.020s user 0.014s sys 0.009s
I20260927 22:00:56.129783 4761 ts_tablet_manager.cc:1428] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641: Time spent starting tablet: real 0.020s user 0.017s sys 0.001s
I20260927 22:00:56.134272 4741 heartbeater.cc:503] Master 127.3.159.62:35273 was elected leader, sending a full tablet report...
I20260927 22:00:56.135761 4762 ts_tablet_manager.cc:1428] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088: Time spent starting tablet: real 0.024s user 0.020s sys 0.004s
W20260927 22:00:56.178215 4742 tablet.cc:2369] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:00:56.241552 4608 tablet.cc:2369] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:00:56.266330 4766 raft_consensus.cc:482] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:00:56.266687 4766 raft_consensus.cc:504] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab2be8055314512980a4eee5e3d9641" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43433 } } peers { permanent_uuid: "1385ac32bdeb44f7ae193ea43980a088" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36081 } } peers { permanent_uuid: "cc918b6fdf6f4f378b5163331ab1ca62" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43061 } }
I20260927 22:00:56.268652 4766 leader_election.cc:288] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1385ac32bdeb44f7ae193ea43980a088 (127.3.159.3:36081), cc918b6fdf6f4f378b5163331ab1ca62 (127.3.159.2:43061)
I20260927 22:00:56.280437 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.269846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:37335 (local address 127.3.159.2:43061)
0927 22:00:56.270230 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:56.270237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:56.270757 (+ 520us) server_negotiation.cc:407] Connection header received
0927 22:00:56.271024 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:56.271062 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:56.271172 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:56.271301 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:00:56.272306 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.273764 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.277522 (+ 3758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.278860 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.279085 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.279174 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.279987 (+ 813us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.280128 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260927 22:00:56.280437 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.269295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:36327 (local address 127.3.159.3:36081)
0927 22:00:56.269940 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:56.269951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:56.271372 (+ 1421us) server_negotiation.cc:407] Connection header received
0927 22:00:56.271496 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:56.271500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:56.271571 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:56.271715 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:00:56.272471 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.273764 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.277526 (+ 3762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.278881 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.279060 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.279174 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.279968 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.280128 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260927 22:00:56.280845 4769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.269541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:43061 (local address 127.3.159.1:37335)
0927 22:00:56.270565 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:56.270600 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:56.270829 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:56.271535 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:56.271539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:56.271560 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:00:56.272044 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:56.272058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.273985 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.273992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:56.277115 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:56.277147 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.279343 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.279352 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:56.279602 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.279728 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 22:00:56.279913 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 22:00:56.280166 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":251,"thread_start_us":544,"threads_started":1}
I20260927 22:00:56.280875 4770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.270044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:36081 (local address 127.3.159.1:36327)
0927 22:00:56.271131 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:56.271154 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:56.271329 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:56.271769 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:56.271773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:56.271799 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:00:56.272273 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:56.272282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.273985 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.273992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:56.277115 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:56.277147 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.279352 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.279358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:56.279602 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.279728 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 22:00:56.279877 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:00:56.280148 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":236,"thread_start_us":557,"threads_started":1}
I20260927 22:00:56.282497 4562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30649c6695914b87bb3af59d7f7adf6c" candidate_uuid: "5ab2be8055314512980a4eee5e3d9641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc918b6fdf6f4f378b5163331ab1ca62" is_pre_election: true
I20260927 22:00:56.282497 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30649c6695914b87bb3af59d7f7adf6c" candidate_uuid: "5ab2be8055314512980a4eee5e3d9641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1385ac32bdeb44f7ae193ea43980a088" is_pre_election: true
I20260927 22:00:56.283210 4696 raft_consensus.cc:2454] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ab2be8055314512980a4eee5e3d9641 in term 0.
I20260927 22:00:56.283210 4562 raft_consensus.cc:2454] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ab2be8055314512980a4eee5e3d9641 in term 0.
I20260927 22:00:56.284631 4361 leader_election.cc:302] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1385ac32bdeb44f7ae193ea43980a088, 5ab2be8055314512980a4eee5e3d9641; no voters:
I20260927 22:00:56.285189 4766 raft_consensus.cc:2781] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:00:56.285486 4766 raft_consensus.cc:482] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:00:56.285614 4766 raft_consensus.cc:3037] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:00:56.288463 4766 raft_consensus.cc:504] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab2be8055314512980a4eee5e3d9641" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43433 } } peers { permanent_uuid: "1385ac32bdeb44f7ae193ea43980a088" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36081 } } peers { permanent_uuid: "cc918b6fdf6f4f378b5163331ab1ca62" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43061 } }
I20260927 22:00:56.289417 4766 leader_election.cc:288] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [CANDIDATE]: Term 1 election: Requested vote from peers 1385ac32bdeb44f7ae193ea43980a088 (127.3.159.3:36081), cc918b6fdf6f4f378b5163331ab1ca62 (127.3.159.2:43061)
I20260927 22:00:56.290174 4562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30649c6695914b87bb3af59d7f7adf6c" candidate_uuid: "5ab2be8055314512980a4eee5e3d9641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc918b6fdf6f4f378b5163331ab1ca62"
I20260927 22:00:56.290184 4696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30649c6695914b87bb3af59d7f7adf6c" candidate_uuid: "5ab2be8055314512980a4eee5e3d9641" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1385ac32bdeb44f7ae193ea43980a088"
I20260927 22:00:56.290453 4696 raft_consensus.cc:3037] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:00:56.290445 4562 raft_consensus.cc:3037] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:00:56.293344 4562 raft_consensus.cc:2454] T 30649c6695914b87bb3af59d7f7adf6c P cc918b6fdf6f4f378b5163331ab1ca62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ab2be8055314512980a4eee5e3d9641 in term 1.
I20260927 22:00:56.293782 4696 raft_consensus.cc:2454] T 30649c6695914b87bb3af59d7f7adf6c P 1385ac32bdeb44f7ae193ea43980a088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ab2be8055314512980a4eee5e3d9641 in term 1.
I20260927 22:00:56.294178 4362 leader_election.cc:302] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ab2be8055314512980a4eee5e3d9641, cc918b6fdf6f4f378b5163331ab1ca62; no voters:
I20260927 22:00:56.294601 4766 raft_consensus.cc:2781] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:00:56.295189 4766 raft_consensus.cc:686] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [term 1 LEADER]: Becoming Leader. State: Replica: 5ab2be8055314512980a4eee5e3d9641, State: Running, Role: LEADER
I20260927 22:00:56.295951 4766 consensus_queue.cc:237] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab2be8055314512980a4eee5e3d9641" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43433 } } peers { permanent_uuid: "1385ac32bdeb44f7ae193ea43980a088" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36081 } } peers { permanent_uuid: "cc918b6fdf6f4f378b5163331ab1ca62" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43061 } }
I20260927 22:00:56.301755 4284 catalog_manager.cc:5495] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ab2be8055314512980a4eee5e3d9641 (127.3.159.1). New cstate: current_term: 1 leader_uuid: "5ab2be8055314512980a4eee5e3d9641" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ab2be8055314512980a4eee5e3d9641" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1385ac32bdeb44f7ae193ea43980a088" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc918b6fdf6f4f378b5163331ab1ca62" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43061 } health_report { overall_health: UNKNOWN } } }
W20260927 22:00:56.351987 4474 tablet.cc:2369] T 30649c6695914b87bb3af59d7f7adf6c P 5ab2be8055314512980a4eee5e3d9641: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 22:00:56 dist-test-slave-sg4j krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:00:56.389811 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.366422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.3.159.1:43433)
0927 22:00:56.366829 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:56.366840 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:56.367334 (+ 494us) server_negotiation.cc:407] Connection header received
0927 22:00:56.367610 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:56.367616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:56.367708 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:56.367838 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:56.369707 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.370622 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.372216 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.372910 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.373106 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.378698 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:56.378729 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:56.378753 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:56.378821 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:56.383342 (+ 4521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.385079 (+ 1737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.385093 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.385096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.385717 (+ 621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.386405 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.386410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.386412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.387915 (+ 1503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:56.388236 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.389108 (+ 872us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.389355 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 22:00:56 dist-test-slave-sg4j krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:00:56 dist-test-slave-sg4j krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546456, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:00:56.449322 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.431689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35584 (local address 127.3.159.1:43433)
0927 22:00:56.432086 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:56.432097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:56.432540 (+ 443us) server_negotiation.cc:407] Connection header received
0927 22:00:56.432963 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:56.432969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:56.433055 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:56.433170 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:56.434720 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.435460 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.436927 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.437715 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.437883 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.442538 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:56.442545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:56.442550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:56.442609 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:56.446063 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.446985 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.446990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.446993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.447161 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.447655 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.447659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.447662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.447990 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:56.448315 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.448603 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.448838 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
Sep 27 22:00:56 dist-test-slave-sg4j krb5kdc[4239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:00:56 dist-test-slave-sg4j krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:00:56.502079 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.484940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35592 (local address 127.3.159.1:43433)
0927 22:00:56.485265 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:56.485273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:56.485727 (+ 454us) server_negotiation.cc:407] Connection header received
0927 22:00:56.485969 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:56.485973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:56.486066 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:56.486182 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:56.488166 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.488916 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.490335 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.490760 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.490902 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.496065 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:56.496072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:56.496076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:56.496122 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:56.499202 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.500021 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.500026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.500030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.500178 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.500577 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.500586 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.500588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.500866 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:56.501138 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.501462 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.501711 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
W20260927 22:00:56.504668 4388 tablet_service.cc:2908] Rejecting scan request for tablet 30649c6695914b87bb3af59d7f7adf6c: Uninitialized: clean time has not yet been initialized
I20260927 22:00:56.508831 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 4345
I20260927 22:00:56.526620 4755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.509942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35602 (local address 127.3.159.1:43433)
0927 22:00:56.510240 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:56.510248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:56.510577 (+ 329us) server_negotiation.cc:407] Connection header received
0927 22:00:56.510754 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:56.510758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:56.510835 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:56.510937 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:56.512328 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.513123 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.515170 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.516125 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.516278 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.519781 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:56.519789 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:56.519793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:56.519850 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:56.522950 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.523996 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.524001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.524004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.524191 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.524753 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.524757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.524760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.525134 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:56.525490 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.525898 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.526165 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:00:56.527628 4468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:00:56.633996 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 4345
I20260927 22:00:56.653878 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 4479
Sep 27 22:00:56 dist-test-slave-sg4j krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:00:56.673858 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.655228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.3.159.2:43061)
0927 22:00:56.655485 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:56.655494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:56.655605 (+ 111us) server_negotiation.cc:407] Connection header received
0927 22:00:56.655870 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:56.655875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:56.655961 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:56.656078 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:56.657494 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.658293 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.659689 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.660115 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.660267 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.665014 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:56.665035 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:56.665058 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:56.665116 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:56.668871 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.670071 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.670084 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.670089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.670305 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.670988 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.670995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.670998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.671678 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:56.672109 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.672982 (+ 873us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.673358 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 22:00:56.675035 4602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:00:56.777324 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 4479
I20260927 22:00:56.796521 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 4613
Sep 27 22:00:56 dist-test-slave-sg4j krb5kdc[4239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:00:56.816177 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:56.797607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.3.159.3:36081)
0927 22:00:56.798274 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:56.798279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:56.798303 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:00:56.798375 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:56.798380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:56.798471 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:56.798587 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:56.800037 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.800890 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.802521 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:56.802971 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:56.803091 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:56.808455 (+ 5364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:56.808477 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:56.808500 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:56.808553 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:56.812081 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.813257 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.813267 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.813270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.813538 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:56.814097 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:56.814101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:56.814104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:56.814658 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:56.814959 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:56.815468 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 22:00:56.815796 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":408,"threads_started":1}
I20260927 22:00:56.817011 4736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:00:56.917961 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 4613
I20260927 22:00:56.938025 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 4252
I20260927 22:00:56.939754 4314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:00:57.067283 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 4252
2026-09-27T22:00:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5081 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:00:57 dist-test-slave-sg4j krb5kdc[4791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:00:57 dist-test-slave-sg4j krb5kdc[4791](info): set up 2 sockets
Sep 27 22:00:57 dist-test-slave-sg4j krb5kdc[4791](info): commencing operation
krb5kdc: starting...
W20260927 22:00:58.182000 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.008s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:00:58 dist-test-slave-sg4j krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:00:58.406335 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:44785
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:44785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:00:58.827185 4804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:00:58.827682 4804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:00:58.827850 4804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:00:58.837018 4804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:00:58.837248 4804 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:00:58.837357 4804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:00:58.837505 4804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:00:58.837598 4804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:00:58.837702 4804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:00:58.850001 4804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:44785
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:44785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:00:58.851836 4804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:00:58.853643 4804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:00:58.868885 4812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:58.868975 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:58.868991 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:00:58.869438 4804 server_base.cc:1034] running on GCE node
I20260927 22:00:58.871169 4804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:00:58.873027 4804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:00:58.874353 4804 hybrid_clock.cc:648] HybridClock initialized: now 1790546458874279 us; error 53 us; skew 500 ppm
I20260927 22:00:58.874821 4804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:00:58 dist-test-slave-sg4j krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:00:58.878584 4804 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:00:58.880560 4804 webserver.cc:460] Webserver started at http://127.3.159.62:43577/ using document root <none> and password file <none>
I20260927 22:00:58.881480 4804 fs_manager.cc:359] Metadata directory not provided
I20260927 22:00:58.881585 4804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:00:58.881958 4804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:00:58.885236 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "227e13530a8646b7a410c2c3188fabef"
format_stamp: "Formatted at 2026-09-27 22:00:58 on dist-test-slave-sg4j"
server_key: "0807cde208ef8deaf3d39693e34df563"
server_key_iv: "e293190232babb9db24890a1872b372b"
server_key_version: "encryptionkey@0"
I20260927 22:00:58.886416 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "227e13530a8646b7a410c2c3188fabef"
format_stamp: "Formatted at 2026-09-27 22:00:58 on dist-test-slave-sg4j"
server_key: "0807cde208ef8deaf3d39693e34df563"
server_key_iv: "e293190232babb9db24890a1872b372b"
server_key_version: "encryptionkey@0"
I20260927 22:00:58.893795 4804 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260927 22:00:58.898928 4819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:58.901005 4804 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:00:58.901252 4804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "227e13530a8646b7a410c2c3188fabef"
format_stamp: "Formatted at 2026-09-27 22:00:58 on dist-test-slave-sg4j"
server_key: "0807cde208ef8deaf3d39693e34df563"
server_key_iv: "e293190232babb9db24890a1872b372b"
server_key_version: "encryptionkey@0"
I20260927 22:00:58.901541 4804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:00:58.931600 4804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:00:58.937783 4804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:00:58.938108 4804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:00:58.966141 4804 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:44785
I20260927 22:00:58.966233 4880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:44785 every 8 connection(s)
I20260927 22:00:58.967976 4804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:00:58.972064 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 4804
I20260927 22:00:58.972345 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:00:58.973234 3708 external_mini_cluster.cc:1428] Setting key 222de7c822c5a7c0d9f9bcb9c967df49
I20260927 22:00:58.974327 4881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:00:58 dist-test-slave-sg4j krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:00:58.993001 4881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef: Bootstrap starting.
I20260927 22:00:58.997372 4881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef: Neither blocks nor log segments found. Creating new log.
I20260927 22:00:58.998739 4881 log.cc:824] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef: Log is configured to *not* fsync() on all Append() calls
I20260927 22:00:59.002022 4884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:58.976293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.3.159.62:44785)
0927 22:00:58.977281 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:58.977296 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:58.977354 (+ 58us) server_negotiation.cc:407] Connection header received
0927 22:00:58.979110 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:58.979142 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:58.979692 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:58.980188 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:58.982523 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:58.984203 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:58.986222 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:58.987128 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:58.987327 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:58.993358 (+ 6031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:58.993368 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:58.993453 (+ 85us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:58.993516 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:58.997057 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:58.998067 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:58.998076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:58.998085 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:58.998279 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:58.998686 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:58.998689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:58.998691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:58.999383 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:58.999736 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:59.000181 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 22:00:59.000651 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":451,"threads_started":1}
I20260927 22:00:59.002625 4881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef: No bootstrap required, opened a new log
I20260927 22:00:59.012475 4881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227e13530a8646b7a410c2c3188fabef" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44785 } }
I20260927 22:00:59.013372 4881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:00:59.013616 4881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 227e13530a8646b7a410c2c3188fabef, State: Initialized, Role: FOLLOWER
I20260927 22:00:59.014469 4881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227e13530a8646b7a410c2c3188fabef" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44785 } }
I20260927 22:00:59.014786 4881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:00:59.014955 4881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:00:59.015161 4881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:00:59.017974 4881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227e13530a8646b7a410c2c3188fabef" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44785 } }
I20260927 22:00:59.018829 4881 leader_election.cc:302] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 227e13530a8646b7a410c2c3188fabef; no voters:
I20260927 22:00:59.019681 4881 leader_election.cc:288] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:00:59.019848 4886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:00:59.020787 4886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [term 1 LEADER]: Becoming Leader. State: Replica: 227e13530a8646b7a410c2c3188fabef, State: Running, Role: LEADER
I20260927 22:00:59.021648 4886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227e13530a8646b7a410c2c3188fabef" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44785 } }
I20260927 22:00:59.022606 4881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:00:59.025624 4887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "227e13530a8646b7a410c2c3188fabef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227e13530a8646b7a410c2c3188fabef" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44785 } } }
I20260927 22:00:59.026356 4887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [sys.catalog]: This master's current role is: LEADER
I20260927 22:00:59.026245 4888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 227e13530a8646b7a410c2c3188fabef. Latest consensus state: current_term: 1 leader_uuid: "227e13530a8646b7a410c2c3188fabef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "227e13530a8646b7a410c2c3188fabef" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44785 } } }
I20260927 22:00:59.026566 4888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef [sys.catalog]: This master's current role is: LEADER
I20260927 22:00:59.034616 4895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:00:59.041539 4895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:00:59.054935 4895 catalog_manager.cc:1347] Generated new cluster ID: ee268eb6bf9149cba3ce6efd3a71887b
I20260927 22:00:59.055085 4895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:00:59.068655 4895 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:00:59.071138 4895 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:00:59.092185 4895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef: Generated new TSK 0
I20260927 22:00:59.093276 4895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:00:59.155676 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:44785
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:00:59.571623 4907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:00:59.572093 4907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:00:59.572266 4907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:00:59.580756 4907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:00:59.580986 4907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:00:59.581148 4907 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:00:59.581301 4907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:00:59.593504 4907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.159.62:44785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:00:59.595348 4907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:00:59.597142 4907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:00:59.614681 4913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:59.614714 4912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:00:59.614749 4915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:00:59.616108 4907 server_base.cc:1034] running on GCE node
I20260927 22:00:59.617157 4907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:00:59.618757 4907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:00:59.620054 4907 hybrid_clock.cc:648] HybridClock initialized: now 1790546459619964 us; error 63 us; skew 500 ppm
I20260927 22:00:59.620482 4907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:00:59 dist-test-slave-sg4j krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:00:59.624912 4907 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:00:59.626945 4907 webserver.cc:460] Webserver started at http://127.3.159.1:39343/ using document root <none> and password file <none>
I20260927 22:00:59.627820 4907 fs_manager.cc:359] Metadata directory not provided
I20260927 22:00:59.627979 4907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:00:59.628319 4907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:00:59.632720 4907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "a47d50bedca44c14bd2feb55080c6436"
format_stamp: "Formatted at 2026-09-27 22:00:59 on dist-test-slave-sg4j"
server_key: "1aee23c759dfa217036a1450e091d1fb"
server_key_iv: "526c53fff25b1475c970be0d4aa03ebc"
server_key_version: "encryptionkey@0"
I20260927 22:00:59.633977 4907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "a47d50bedca44c14bd2feb55080c6436"
format_stamp: "Formatted at 2026-09-27 22:00:59 on dist-test-slave-sg4j"
server_key: "1aee23c759dfa217036a1450e091d1fb"
server_key_iv: "526c53fff25b1475c970be0d4aa03ebc"
server_key_version: "encryptionkey@0"
I20260927 22:00:59.640679 4907 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:00:59.645673 4922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:59.647763 4907 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:00:59.648020 4907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "a47d50bedca44c14bd2feb55080c6436"
format_stamp: "Formatted at 2026-09-27 22:00:59 on dist-test-slave-sg4j"
server_key: "1aee23c759dfa217036a1450e091d1fb"
server_key_iv: "526c53fff25b1475c970be0d4aa03ebc"
server_key_version: "encryptionkey@0"
I20260927 22:00:59.648300 4907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:00:59.684700 4907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:00:59.691522 4907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:00:59.691902 4907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:00:59.695592 4907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:00:59.695744 4907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:00:59.695960 4907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:00:59.696058 4907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:00:59 dist-test-slave-sg4j krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546459, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:00:59.731899 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:59.697145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:41545 (local address 127.3.159.62:44785)
0927 22:00:59.697859 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:59.697869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:59.699705 (+ 1836us) server_negotiation.cc:407] Connection header received
0927 22:00:59.703478 (+ 3773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:59.703490 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:59.703649 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:59.703852 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:59.708196 (+ 4344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:59.709621 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:59.712376 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:59.713443 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:59.713658 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:59.720898 (+ 7240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:59.720911 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:59.720920 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:59.720999 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:59.726404 (+ 5405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:59.727872 (+ 1468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:59.727883 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:59.727889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:59.728106 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:59.728972 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:59.728979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:59.728982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:59.729507 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:59.729684 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:59.730845 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0927 22:00:59.731260 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":374,"threads_started":1}
I20260927 22:00:59.732533 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:59.698026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44785 (local address 127.3.159.1:41545)
0927 22:00:59.699242 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:59.699456 (+ 214us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:59.702571 (+ 3115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:59.704585 (+ 2014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:59.704614 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:59.705997 (+ 1383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:00:59.707698 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:59.707753 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:59.709954 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:59.709959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:59.712085 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:59.712117 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:59.714043 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:59.714062 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:59.714362 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:59.715754 (+ 1392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:00:59.715790 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:00:59.720219 (+ 4429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:00:59.726785 (+ 6566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:59.726804 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:59.726858 (+ 54us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:59.727688 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:59.728293 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:59.728303 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:59.728308 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:59.728756 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:59.729783 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:00:59.729793 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:00:59.730280 (+ 487us) client_negotiation.cc:769] Sending connection context
0927 22:00:59.730606 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 22:00:59.731115 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":531,"threads_started":1}
I20260927 22:00:59.768182 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:59.742510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:34535 (local address 127.3.159.62:44785)
0927 22:00:59.742880 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0927 22:00:59.742891 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:00:59.745748 (+ 2857us) server_negotiation.cc:407] Connection header received
0927 22:00:59.746077 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:00:59.746082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:00:59.746188 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:00:59.746312 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:00:59.748382 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:59.749243 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:59.752793 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:00:59.753741 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:00:59.753905 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:59.758260 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:00:59.758267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:00:59.758272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:00:59.758324 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:00:59.762193 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:59.763776 (+ 1583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:59.763781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:59.763784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:59.763991 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:00:59.764797 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:00:59.764805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:00:59.764809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:00:59.765466 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:00:59.765995 (+ 529us) server_negotiation.cc:1035] Waiting for connection context
0927 22:00:59.767541 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0927 22:00:59.767879 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 22:00:59.768289 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:00:59.743707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44785 (local address 127.3.159.1:34535)
0927 22:00:59.745488 (+ 1781us) negotiation.cc:107] Waiting for socket to connect
0927 22:00:59.745527 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 22:00:59.745849 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:00:59.746422 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:00:59.746430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:00:59.747401 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:00:59.748024 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:59.748046 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:59.749673 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:59.749683 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:59.752318 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:00:59.752359 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:00:59.754098 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:00:59.754110 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:00:59.754425 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:00:59.755759 (+ 1334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:00:59.755792 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:00:59.757713 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:00:59.762564 (+ 4851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:59.762571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:59.762576 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:59.763536 (+ 960us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:59.764183 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:00:59.764193 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:00:59.764201 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:00:59.764488 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:00:59.765662 (+ 1174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:00:59.765672 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:00:59.766465 (+ 793us) client_negotiation.cc:769] Sending connection context
0927 22:00:59.766731 (+ 266us) client_negotiation.cc:240] Negotiation successful
0927 22:00:59.767424 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1460,"thread_start_us":1061,"threads_started":1}
I20260927 22:00:59.776088 4907 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:34029
I20260927 22:00:59.776113 5042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:34029 every 8 connection(s)
I20260927 22:00:59.777797 4907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:00:59.781255 5043 heartbeater.cc:347] Connected to a master server at 127.3.159.62:44785
I20260927 22:00:59.781670 5043 heartbeater.cc:464] Registering TS with master...
I20260927 22:00:59.782552 5043 heartbeater.cc:511] Master 127.3.159.62:44785 requested a full tablet report, sending...
I20260927 22:00:59.783674 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 4907
I20260927 22:00:59.783917 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:00:59.784682 3708 external_mini_cluster.cc:1428] Setting key 30c409ed73f5883d29403e7acabbfbd1
I20260927 22:00:59.785928 4836 ts_manager.cc:194] Registered new tserver with Master: a47d50bedca44c14bd2feb55080c6436 (127.3.159.1:34029)
I20260927 22:00:59.787941 4836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:41545
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:00:59.839246 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:44785
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:00.263837 5047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:00.264302 5047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:00.264480 5047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:00.273015 5047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:00.273245 5047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:00.273497 5047 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:01:00.273645 5047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:01:00.285560 5047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.159.62:44785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:00.287400 5047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:00.289204 5047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:00.308511 5055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:00.308607 5052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:00.308488 5053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:00.310134 5047 server_base.cc:1034] running on GCE node
I20260927 22:01:00.311008 5047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:00.312345 5047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:00.313652 5047 hybrid_clock.cc:648] HybridClock initialized: now 1790546460313571 us; error 57 us; skew 500 ppm
I20260927 22:01:00.314069 5047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:00 dist-test-slave-sg4j krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546460, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:00.318625 5047 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:01:00.320832 5047 webserver.cc:460] Webserver started at http://127.3.159.2:45877/ using document root <none> and password file <none>
I20260927 22:01:00.321852 5047 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:00.322005 5047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:00.322439 5047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:00.326404 5047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "4bbfb3f083be48e99a813e3939b324a5"
format_stamp: "Formatted at 2026-09-27 22:01:00 on dist-test-slave-sg4j"
server_key: "4cb6326ed5a4904f3766af8f3e8ad845"
server_key_iv: "3473ca8fe97adbebb999ffe4141a4b26"
server_key_version: "encryptionkey@0"
I20260927 22:01:00.327502 5047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "4bbfb3f083be48e99a813e3939b324a5"
format_stamp: "Formatted at 2026-09-27 22:01:00 on dist-test-slave-sg4j"
server_key: "4cb6326ed5a4904f3766af8f3e8ad845"
server_key_iv: "3473ca8fe97adbebb999ffe4141a4b26"
server_key_version: "encryptionkey@0"
I20260927 22:01:00.335259 5047 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 22:01:00.340745 5062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:00.343357 5047 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:01:00.343688 5047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "4bbfb3f083be48e99a813e3939b324a5"
format_stamp: "Formatted at 2026-09-27 22:01:00 on dist-test-slave-sg4j"
server_key: "4cb6326ed5a4904f3766af8f3e8ad845"
server_key_iv: "3473ca8fe97adbebb999ffe4141a4b26"
server_key_version: "encryptionkey@0"
I20260927 22:01:00.344007 5047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:00.363626 5047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:00.369777 5047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:00.370172 5047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:00.373857 5047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:00.374008 5047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:00.374271 5047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:00.374374 5047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:01:00 dist-test-slave-sg4j krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546460, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:00.408010 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:00.375143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:51995 (local address 127.3.159.62:44785)
0927 22:01:00.375676 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:00.375681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:00.377881 (+ 2200us) server_negotiation.cc:407] Connection header received
0927 22:01:00.381545 (+ 3664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:00.381555 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:00.381738 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:00.381902 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:00.385524 (+ 3622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:00.387300 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:00.390526 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:00.391528 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:00.391768 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:00.398067 (+ 6299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:00.398073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:00.398076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:00.398116 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:00.403114 (+ 4998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:00.404395 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:00.404406 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:00.404413 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:00.404677 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:00.405522 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:00.405557 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:00.405562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:00.406031 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:00.406223 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:00.407265 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0927 22:01:00.407588 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":327,"threads_started":1}
I20260927 22:01:00.409348 5072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:00.376255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44785 (local address 127.3.159.2:51995)
0927 22:01:00.377575 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:00.377651 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:00.380736 (+ 3085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:00.382452 (+ 1716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:00.382476 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:00.383849 (+ 1373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:00.385053 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:00.385114 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:00.387619 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:00.387634 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:00.390137 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:00.390215 (+ 78us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:00.392013 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:00.392029 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:00.392330 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:00.393549 (+ 1219us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:00.393580 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:00.397748 (+ 4168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:00.403445 (+ 5697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:00.403456 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:00.403474 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:00.404173 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:00.404894 (+ 721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:00.404902 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:00.404907 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:00.405299 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:00.406199 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:00.406208 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:00.406821 (+ 613us) client_negotiation.cc:769] Sending connection context
0927 22:01:00.407156 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 22:01:00.407759 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":517,"threads_started":1}
I20260927 22:01:00.441952 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:00.419899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44785 (local address 127.3.159.2:49795)
0927 22:01:00.420928 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:00.420966 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:00.421303 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:00.421952 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:00.421962 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:00.422970 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:00.423905 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:00.423932 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:00.425482 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:00.425493 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:00.428462 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:00.428495 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:00.429619 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:00.429631 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:00.429926 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:00.430922 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:00.430962 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:00.432695 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:00.437409 (+ 4714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:00.437421 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:00.437428 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:00.438445 (+ 1017us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:00.439066 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:00.439073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:00.439079 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:00.439363 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:00.440569 (+ 1206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:00.440574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:00.440813 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 22:01:00.440954 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:01:00.441297 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":495,"threads_started":1}
I20260927 22:01:00.441998 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:00.419344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:49795 (local address 127.3.159.62:44785)
0927 22:01:00.419712 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:00.419723 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:00.421188 (+ 1465us) server_negotiation.cc:407] Connection header received
0927 22:01:00.421616 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:00.421621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:00.421759 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:00.421861 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:00.424302 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:00.425227 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:00.428788 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:00.429265 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:00.429496 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:00.433067 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:00.433074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:00.433078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:00.433127 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:00.437120 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:00.438696 (+ 1576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:00.438705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:00.438711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:00.438899 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:00.439594 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:00.439604 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:00.439610 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:00.440358 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:00.440587 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:00.441128 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 22:01:00.441521 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260927 22:01:00.446949 5182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:33313 every 8 connection(s)
I20260927 22:01:00.446943 5047 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:33313
I20260927 22:01:00.449218 5047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:01:00.452054 5183 heartbeater.cc:347] Connected to a master server at 127.3.159.62:44785
I20260927 22:01:00.452386 5183 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:00.453348 5183 heartbeater.cc:511] Master 127.3.159.62:44785 requested a full tablet report, sending...
I20260927 22:01:00.455231 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 5047
I20260927 22:01:00.455457 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:01:00.455443 4836 ts_manager.cc:194] Registered new tserver with Master: 4bbfb3f083be48e99a813e3939b324a5 (127.3.159.2:33313)
I20260927 22:01:00.455984 3708 external_mini_cluster.cc:1428] Setting key 669c1844ff8eba651d4c85a514a0f26f
I20260927 22:01:00.456441 4836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:51995
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:01:00.502739 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:44785
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:37721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:00.792971 5043 heartbeater.cc:503] Master 127.3.159.62:44785 was elected leader, sending a full tablet report...
W20260927 22:01:00.896605 5187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:00.896944 5187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:00.897065 5187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:00.907536 5187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:00.907790 5187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:00.907951 5187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:01:00.908102 5187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:01:00.921626 5187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:37721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.159.62:44785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:00.923427 5187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:00.925043 5187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:00.941784 5192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:00.941829 5193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:00.941720 5195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:00.942847 5187 server_base.cc:1034] running on GCE node
I20260927 22:01:00.943600 5187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:00.944793 5187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:00.946238 5187 hybrid_clock.cc:648] HybridClock initialized: now 1790546460946037 us; error 149 us; skew 500 ppm
I20260927 22:01:00.946651 5187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:00 dist-test-slave-sg4j krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546460, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:00.955099 5187 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:01:00.957006 5187 webserver.cc:460] Webserver started at http://127.3.159.3:36311/ using document root <none> and password file <none>
I20260927 22:01:00.957911 5187 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:00.958055 5187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:00.958410 5187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:00.961656 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "80f3b419e0044f16b72d697363b9c1ea"
format_stamp: "Formatted at 2026-09-27 22:01:00 on dist-test-slave-sg4j"
server_key: "6bd9ee69d7d4127017ca01061f0d26fd"
server_key_iv: "649101827b15175f3f5ebc8138e058a2"
server_key_version: "encryptionkey@0"
I20260927 22:01:00.962566 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "80f3b419e0044f16b72d697363b9c1ea"
format_stamp: "Formatted at 2026-09-27 22:01:00 on dist-test-slave-sg4j"
server_key: "6bd9ee69d7d4127017ca01061f0d26fd"
server_key_iv: "649101827b15175f3f5ebc8138e058a2"
server_key_version: "encryptionkey@0"
I20260927 22:01:00.968511 5187 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 22:01:00.972659 5202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:00.974764 5187 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:01:00.975023 5187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "80f3b419e0044f16b72d697363b9c1ea"
format_stamp: "Formatted at 2026-09-27 22:01:00 on dist-test-slave-sg4j"
server_key: "6bd9ee69d7d4127017ca01061f0d26fd"
server_key_iv: "649101827b15175f3f5ebc8138e058a2"
server_key_version: "encryptionkey@0"
I20260927 22:01:00.975332 5187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:00.995537 5187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:01.000511 5187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:01.000834 5187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:01.004452 5187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:01.004597 5187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:01.004801 5187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:01.004881 5187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:01:01 dist-test-slave-sg4j krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546460, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:01.042979 5212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.006487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44785 (local address 127.3.159.3:50265)
0927 22:01:01.007647 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:01.007702 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:01.010784 (+ 3082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:01.012237 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:01.012260 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:01.013554 (+ 1294us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:01.015118 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.015183 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.016898 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.016907 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.021439 (+ 4532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.021467 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.023349 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.023357 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.023652 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.024664 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:01.024695 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:01.028925 (+ 4230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:01.034662 (+ 5737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:01.034682 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:01.034724 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:01.035635 (+ 911us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:01.036438 (+ 803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:01.036443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:01.036446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:01.036820 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:01.037891 (+ 1071us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:01.037904 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:01.038773 (+ 869us) client_negotiation.cc:769] Sending connection context
0927 22:01:01.039315 (+ 542us) client_negotiation.cc:240] Negotiation successful
0927 22:01:01.040581 (+ 1266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":497,"threads_started":1}
I20260927 22:01:01.043200 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.005781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:50265 (local address 127.3.159.62:44785)
0927 22:01:01.006573 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.006578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.007881 (+ 1303us) server_negotiation.cc:407] Connection header received
0927 22:01:01.011381 (+ 3500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.011387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.011510 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.011690 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:01.015577 (+ 3887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.016609 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.021843 (+ 5234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.022651 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.022881 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.029536 (+ 6655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:01.029543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:01.029548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:01.029600 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:01.034334 (+ 4734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:01.035911 (+ 1577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:01.035922 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:01.035929 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:01.036190 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:01.037051 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:01.037057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:01.037059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:01.037570 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:01.037778 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.041924 (+ 4146us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.042553 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":550,"threads_started":1}
I20260927 22:01:01.070858 5187 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:44003
I20260927 22:01:01.070982 5322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:44003 every 8 connection(s)
I20260927 22:01:01.072888 5187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:01:01.076488 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 5187
I20260927 22:01:01.076761 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:01:01.077456 3708 external_mini_cluster.cc:1428] Setting key 41f3c443fdfe385a3de02b2c35270cd7
I20260927 22:01:01.078234 5323 heartbeater.cc:347] Connected to a master server at 127.3.159.62:44785
I20260927 22:01:01.078588 5323 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:01.079144 5299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.054664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44785 (local address 127.3.159.3:56617)
0927 22:01:01.058483 (+ 3819us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:01.058519 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:01.058728 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:01.059561 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:01.059571 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:01.060781 (+ 1210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:01.061605 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.061640 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.063965 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.063976 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.066851 (+ 2875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.066892 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.068374 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.068382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.068589 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.069411 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:01.069446 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:01.070911 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:01.075039 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:01.075047 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:01.075052 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:01.075862 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:01.076417 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:01.076425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:01.076430 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:01.076638 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:01.077651 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:01.077656 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:01.077904 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 22:01:01.078085 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 22:01:01.078569 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3449,"thread_start_us":3212,"threads_started":1}
I20260927 22:01:01.079162 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.054634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:56617 (local address 127.3.159.62:44785)
0927 22:01:01.054961 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.054970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.058713 (+ 3743us) server_negotiation.cc:407] Connection header received
0927 22:01:01.058888 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.058898 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.059043 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.059224 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:01.062062 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.063660 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.067278 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.068074 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.068268 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.071391 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:01.071399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:01.071404 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:01.071452 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:01.074827 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:01.076097 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:01.076101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:01.076104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:01.076262 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:01.076837 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:01.076846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:01.076853 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:01.077329 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:01.077568 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.078366 (+ 798us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.078697 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 22:01:01.079838 5323 heartbeater.cc:511] Master 127.3.159.62:44785 requested a full tablet report, sending...
I20260927 22:01:01.081993 4834 ts_manager.cc:194] Registered new tserver with Master: 80f3b419e0044f16b72d697363b9c1ea (127.3.159.3:44003)
I20260927 22:01:01.083470 4834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:50265
I20260927 22:01:01.096834 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:01:01 dist-test-slave-sg4j krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546461, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:01:01 dist-test-slave-sg4j krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546461, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:01.162272 5211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.139856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.3.159.62:44785)
0927 22:01:01.140055 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.140060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.140336 (+ 276us) server_negotiation.cc:407] Connection header received
0927 22:01:01.140645 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.140651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.140768 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.140907 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:01.143145 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.144069 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.146004 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.146682 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.146853 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.154394 (+ 7541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:01.154407 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:01.154413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:01.154465 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:01.157995 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:01.159153 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:01.159162 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:01.159165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:01.159372 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:01.160144 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:01.160152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:01.160155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:01.160648 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:01.160987 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.161573 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.161827 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 22:01:01.168445 4834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:01:01.176388 4834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:01.208047 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.195550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.3.159.2:33313)
0927 22:01:01.197939 (+ 2389us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.197948 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.197981 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:01:01.198105 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.198117 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.198423 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.199565 (+ 1142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:01.199772 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.201311 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.204668 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.206193 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.206686 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.206982 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.207393 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.207614 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2223,"thread_start_us":2101,"threads_started":1}
I20260927 22:01:01.208110 5337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.194789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:33313 (local address 127.0.0.1:35678)
0927 22:01:01.196492 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:01.196540 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:01.196751 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:01.198713 (+ 1962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:01.198723 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:01.198760 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:01.199346 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.199359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.201630 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.201636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.204455 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.204469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.206789 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.206798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.207065 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.207137 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 22:01:01.207324 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 22:01:01.207592 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1369,"mutex_wait_us":7,"thread_start_us":2936,"threads_started":1}
I20260927 22:01:01.211251 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.198693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.3.159.1:34029)
0927 22:01:01.200200 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.200216 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.201056 (+ 840us) server_negotiation.cc:407] Connection header received
0927 22:01:01.201249 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.201258 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.201540 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.201780 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:01.202744 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.204290 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.207563 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.209331 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.209511 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.209763 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.210704 (+ 941us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.210905 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1299,"thread_start_us":1120,"threads_started":1}
I20260927 22:01:01.211630 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.195849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.3.159.3:44003)
0927 22:01:01.197542 (+ 1693us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.197552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.198424 (+ 872us) server_negotiation.cc:407] Connection header received
0927 22:01:01.199326 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.199336 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.199598 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.199760 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:01.201031 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.203013 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.206484 (+ 3471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.209824 (+ 3340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.210220 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.210457 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.211091 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.211275 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1482,"thread_start_us":1274,"threads_started":1}
I20260927 22:01:01.212199 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.194924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:44003 (local address 127.0.0.1:37154)
0927 22:01:01.198306 (+ 3382us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:01.198327 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:01.199082 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:01.200170 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:01.200174 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:01.200201 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:01.200736 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.200749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.203246 (+ 2497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.203252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.206215 (+ 2963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.206233 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.210623 (+ 4390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.210633 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.210863 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.210901 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:01:01.211020 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 22:01:01.211215 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3266,"mutex_wait_us":3,"thread_start_us":1200,"threads_started":1}
I20260927 22:01:01.212216 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.199054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:34029 (local address 127.0.0.1:39692)
0927 22:01:01.200838 (+ 1784us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:01.200884 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:01.201061 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:01.201774 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:01.201779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:01.201811 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:01.202317 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.202332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.204493 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.204503 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.207304 (+ 2801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.207326 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.210195 (+ 2869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.210203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.210396 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.210444 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:01:01.211592 (+ 1148us) client_negotiation.cc:240] Negotiation successful
0927 22:01:01.211791 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1640,"mutex_wait_us":30,"thread_start_us":1409,"threads_started":1}
I20260927 22:01:01.215237 5113 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5b3e36ff7b426baf90f38d27d69751 (DEFAULT_TABLE table=test-table [id=23ba4fb8f9f8495d988c52d819e2ad37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:01.217538 5113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5b3e36ff7b426baf90f38d27d69751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:01.218240 4973 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5b3e36ff7b426baf90f38d27d69751 (DEFAULT_TABLE table=test-table [id=23ba4fb8f9f8495d988c52d819e2ad37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:01.218550 5253 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5b3e36ff7b426baf90f38d27d69751 (DEFAULT_TABLE table=test-table [id=23ba4fb8f9f8495d988c52d819e2ad37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:01.220103 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5b3e36ff7b426baf90f38d27d69751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:01.220371 5253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c5b3e36ff7b426baf90f38d27d69751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:01.234025 5342 tablet_bootstrap.cc:492] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: Bootstrap starting.
I20260927 22:01:01.240258 5342 tablet_bootstrap.cc:654] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:01.242084 5342 log.cc:824] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:01.243942 5344 tablet_bootstrap.cc:492] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: Bootstrap starting.
I20260927 22:01:01.246910 5343 tablet_bootstrap.cc:492] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5: Bootstrap starting.
I20260927 22:01:01.251286 5344 tablet_bootstrap.cc:654] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:01.252272 5343 tablet_bootstrap.cc:654] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:01.253402 5344 log.cc:824] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:01.254138 5342 tablet_bootstrap.cc:492] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: No bootstrap required, opened a new log
I20260927 22:01:01.254364 5343 log.cc:824] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:01.254614 5342 ts_tablet_manager.cc:1397] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.011s
I20260927 22:01:01.257746 5344 tablet_bootstrap.cc:492] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: No bootstrap required, opened a new log
I20260927 22:01:01.258153 5344 ts_tablet_manager.cc:1397] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260927 22:01:01.259387 5343 tablet_bootstrap.cc:492] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5: No bootstrap required, opened a new log
I20260927 22:01:01.260872 5343 ts_tablet_manager.cc:1397] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260927 22:01:01.269074 5344 raft_consensus.cc:348] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } }
I20260927 22:01:01.269771 5344 raft_consensus.cc:374] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:01.269335 5342 raft_consensus.cc:348] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } }
I20260927 22:01:01.269891 5344 raft_consensus.cc:729] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f3b419e0044f16b72d697363b9c1ea, State: Initialized, Role: FOLLOWER
I20260927 22:01:01.269975 5342 raft_consensus.cc:374] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:01.270109 5342 raft_consensus.cc:729] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47d50bedca44c14bd2feb55080c6436, State: Initialized, Role: FOLLOWER
I20260927 22:01:01.271332 5344 consensus_queue.cc:260] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } }
I20260927 22:01:01.271327 5342 consensus_queue.cc:260] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } }
I20260927 22:01:01.274122 5323 heartbeater.cc:503] Master 127.3.159.62:44785 was elected leader, sending a full tablet report...
I20260927 22:01:01.274654 5344 ts_tablet_manager.cc:1428] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260927 22:01:01.274989 5343 raft_consensus.cc:348] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } }
I20260927 22:01:01.275486 5343 raft_consensus.cc:374] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:01.275573 5343 raft_consensus.cc:729] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbfb3f083be48e99a813e3939b324a5, State: Initialized, Role: FOLLOWER
I20260927 22:01:01.276561 5343 consensus_queue.cc:260] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } }
I20260927 22:01:01.278669 5342 ts_tablet_manager.cc:1428] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: Time spent starting tablet: real 0.024s user 0.015s sys 0.009s
I20260927 22:01:01.280265 5183 heartbeater.cc:503] Master 127.3.159.62:44785 was elected leader, sending a full tablet report...
W20260927 22:01:01.281510 5044 tablet.cc:2369] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:01.282521 5343 ts_tablet_manager.cc:1428] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5: Time spent starting tablet: real 0.021s user 0.023s sys 0.000s
I20260927 22:01:01.283692 5349 raft_consensus.cc:482] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:01.284093 5349 raft_consensus.cc:504] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } }
I20260927 22:01:01.287338 5349 leader_election.cc:288] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80f3b419e0044f16b72d697363b9c1ea (127.3.159.3:44003), 4bbfb3f083be48e99a813e3939b324a5 (127.3.159.2:33313)
I20260927 22:01:01.297256 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.287948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:49449 (local address 127.3.159.3:44003)
0927 22:01:01.288167 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.288173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.289470 (+ 1297us) server_negotiation.cc:407] Connection header received
0927 22:01:01.289838 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.289844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.289923 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.289989 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:01.291112 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.292585 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.295235 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.296067 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.296240 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.296367 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.296779 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.296923 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 22:01:01.297569 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.287900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:33313 (local address 127.3.159.1:33851)
0927 22:01:01.289825 (+ 1925us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:01.289844 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:01.290025 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:01.290394 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:01.290399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:01.290466 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:01.291002 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.291011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.292141 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.292147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.294943 (+ 2796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.294971 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.296406 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.296413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.296625 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.296653 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:01:01.296777 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 22:01:01.296984 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1775,"mutex_wait_us":13,"spinlock_wait_cycles":5248,"thread_start_us":1572,"threads_started":1}
I20260927 22:01:01.297317 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.288834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:33851 (local address 127.3.159.2:33313)
0927 22:01:01.289267 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.289274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.289952 (+ 678us) server_negotiation.cc:407] Connection header received
0927 22:01:01.290182 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.290188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.290270 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.290375 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:01.291180 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.291985 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.295203 (+ 3218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.296091 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.296290 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.296447 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.296811 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.296980 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266}
I20260927 22:01:01.297766 5351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.287461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:44003 (local address 127.3.159.1:49449)
0927 22:01:01.289272 (+ 1811us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:01.289299 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:01.289612 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:01.290218 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:01.290223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:01.290251 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:01.290800 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.290919 (+ 119us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.292816 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.292822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.295008 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.295029 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.296349 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.296357 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.296589 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.296629 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:01:01.296752 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:01:01.296949 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1643,"mutex_wait_us":83,"spinlock_wait_cycles":33664,"thread_start_us":2624,"threads_started":1}
I20260927 22:01:01.299279 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5b3e36ff7b426baf90f38d27d69751" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bbfb3f083be48e99a813e3939b324a5" is_pre_election: true
I20260927 22:01:01.299525 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5b3e36ff7b426baf90f38d27d69751" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f3b419e0044f16b72d697363b9c1ea" is_pre_election: true
I20260927 22:01:01.300019 5133 raft_consensus.cc:2454] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47d50bedca44c14bd2feb55080c6436 in term 0.
I20260927 22:01:01.300431 5273 raft_consensus.cc:2454] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47d50bedca44c14bd2feb55080c6436 in term 0.
I20260927 22:01:01.301069 4925 leader_election.cc:302] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbfb3f083be48e99a813e3939b324a5, a47d50bedca44c14bd2feb55080c6436; no voters:
I20260927 22:01:01.301695 5349 raft_consensus.cc:2781] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:01.301882 5349 raft_consensus.cc:482] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:01.302032 5349 raft_consensus.cc:3037] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:01.305886 5349 raft_consensus.cc:504] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } }
I20260927 22:01:01.306783 5349 leader_election.cc:288] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 election: Requested vote from peers 80f3b419e0044f16b72d697363b9c1ea (127.3.159.3:44003), 4bbfb3f083be48e99a813e3939b324a5 (127.3.159.2:33313)
I20260927 22:01:01.307621 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5b3e36ff7b426baf90f38d27d69751" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f3b419e0044f16b72d697363b9c1ea"
I20260927 22:01:01.307628 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5b3e36ff7b426baf90f38d27d69751" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bbfb3f083be48e99a813e3939b324a5"
I20260927 22:01:01.307929 5273 raft_consensus.cc:3037] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:01.308009 5133 raft_consensus.cc:3037] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:01.310994 5273 raft_consensus.cc:2454] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47d50bedca44c14bd2feb55080c6436 in term 1.
I20260927 22:01:01.311100 5133 raft_consensus.cc:2454] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47d50bedca44c14bd2feb55080c6436 in term 1.
I20260927 22:01:01.311910 4925 leader_election.cc:302] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4bbfb3f083be48e99a813e3939b324a5, 80f3b419e0044f16b72d697363b9c1ea, a47d50bedca44c14bd2feb55080c6436; no voters:
I20260927 22:01:01.312405 5349 raft_consensus.cc:2781] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:01.313180 5349 raft_consensus.cc:686] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 1 LEADER]: Becoming Leader. State: Replica: a47d50bedca44c14bd2feb55080c6436, State: Running, Role: LEADER
I20260927 22:01:01.314236 5349 consensus_queue.cc:237] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } }
I20260927 22:01:01.320551 4836 catalog_manager.cc:5495] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47d50bedca44c14bd2feb55080c6436 (127.3.159.1). New cstate: current_term: 1 leader_uuid: "a47d50bedca44c14bd2feb55080c6436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } health_report { overall_health: HEALTHY } } }
W20260927 22:01:01.325544 5324 tablet.cc:2369] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:01.360980 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.351475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39694 (local address 127.3.159.1:34029)
0927 22:01:01.351721 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.351732 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.351999 (+ 267us) server_negotiation.cc:407] Connection header received
0927 22:01:01.352530 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.352536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.353696 (+ 1160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.353896 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:01.354574 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.355583 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.357573 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.358065 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.358204 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.359008 (+ 804us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:01.359962 (+ 954us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:01.360300 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.360626 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.360725 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 22:01:01.372449 5273 raft_consensus.cc:1261] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea [term 1 FOLLOWER]: Refusing update from remote peer a47d50bedca44c14bd2feb55080c6436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:01.372449 5133 raft_consensus.cc:1261] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5 [term 1 FOLLOWER]: Refusing update from remote peer a47d50bedca44c14bd2feb55080c6436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:01.374375 5349 consensus_queue.cc:1035] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:01.375018 5353 consensus_queue.cc:1035] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:01.418375 4834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:01.418807 4834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:01.423030 4834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef: Sending DeleteTablet for 3 replicas of tablet 5c5b3e36ff7b426baf90f38d27d69751
I20260927 22:01:01.424743 5253 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5b3e36ff7b426baf90f38d27d69751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:01 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:37154
I20260927 22:01:01.424834 5113 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5b3e36ff7b426baf90f38d27d69751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:01 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:35678
I20260927 22:01:01.425621 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5b3e36ff7b426baf90f38d27d69751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:01 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:39692
I20260927 22:01:01.431357 4834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:01:01.434489 5367 tablet_replica.cc:330] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5: stopping tablet replica
I20260927 22:01:01.435395 5367 raft_consensus.cc:2229] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:01.436192 5367 raft_consensus.cc:2258] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 22:01:01.436393 4834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:01.438625 5367 ts_tablet_manager.cc:1905] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:01.445541 5366 tablet_replica.cc:330] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: stopping tablet replica
I20260927 22:01:01.446133 5366 raft_consensus.cc:2229] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:01.447154 5366 raft_consensus.cc:2258] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:01.450133 5366 ts_tablet_manager.cc:1905] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:01.452574 5367 ts_tablet_manager.cc:1918] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:01.452805 5367 log.cc:1196] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/5c5b3e36ff7b426baf90f38d27d69751
I20260927 22:01:01.453553 5367 ts_tablet_manager.cc:1939] T 5c5b3e36ff7b426baf90f38d27d69751 P 4bbfb3f083be48e99a813e3939b324a5: Deleting consensus metadata
I20260927 22:01:01.455888 4822 catalog_manager.cc:4841] TS 4bbfb3f083be48e99a813e3939b324a5 (127.3.159.2:33313): tablet 5c5b3e36ff7b426baf90f38d27d69751 (table test-table [id=23ba4fb8f9f8495d988c52d819e2ad37]) successfully deleted
I20260927 22:01:01.460250 5366 ts_tablet_manager.cc:1918] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:01.460682 5366 log.cc:1196] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/5c5b3e36ff7b426baf90f38d27d69751
I20260927 22:01:01.460270 5368 tablet_replica.cc:330] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: stopping tablet replica
I20260927 22:01:01.461359 5368 raft_consensus.cc:2229] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:01:01.461627 5366 ts_tablet_manager.cc:1939] T 5c5b3e36ff7b426baf90f38d27d69751 P 80f3b419e0044f16b72d697363b9c1ea: Deleting consensus metadata
I20260927 22:01:01.462177 5368 raft_consensus.cc:2258] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:01.465339 5113 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9c82df14304d28967184dd1f59fc7b (DEFAULT_TABLE table=test-table [id=dbea8caada4547cdb7114c7046b7f425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:01.465775 4973 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9c82df14304d28967184dd1f59fc7b (DEFAULT_TABLE table=test-table [id=dbea8caada4547cdb7114c7046b7f425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:01.465909 5253 tablet_service.cc:1467] Processing CreateTablet for tablet 7a9c82df14304d28967184dd1f59fc7b (DEFAULT_TABLE table=test-table [id=dbea8caada4547cdb7114c7046b7f425]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:01.466205 5113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9c82df14304d28967184dd1f59fc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:01.466578 5253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9c82df14304d28967184dd1f59fc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:01.466586 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a9c82df14304d28967184dd1f59fc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:01.475106 4821 catalog_manager.cc:4841] TS 80f3b419e0044f16b72d697363b9c1ea (127.3.159.3:44003): tablet 5c5b3e36ff7b426baf90f38d27d69751 (table test-table [id=23ba4fb8f9f8495d988c52d819e2ad37]) successfully deleted
I20260927 22:01:01.475714 5344 tablet_bootstrap.cc:492] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea: Bootstrap starting.
I20260927 22:01:01.476629 5342 tablet_bootstrap.cc:492] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436: Bootstrap starting.
I20260927 22:01:01.477769 5344 tablet_bootstrap.cc:654] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:01.478682 5342 tablet_bootstrap.cc:654] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:01.481078 5344 tablet_bootstrap.cc:492] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea: No bootstrap required, opened a new log
I20260927 22:01:01.481503 5344 ts_tablet_manager.cc:1397] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 22:01:01.483469 5343 tablet_bootstrap.cc:492] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5: Bootstrap starting.
I20260927 22:01:01.484913 5342 tablet_bootstrap.cc:492] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436: No bootstrap required, opened a new log
I20260927 22:01:01.485119 5342 ts_tablet_manager.cc:1397] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260927 22:01:01.485273 5368 ts_tablet_manager.cc:1905] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:01.485889 5343 tablet_bootstrap.cc:654] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:01.482641 5344 raft_consensus.cc:348] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.486270 5342 raft_consensus.cc:348] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.486472 5344 raft_consensus.cc:374] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:01.486549 5342 raft_consensus.cc:374] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:01.486647 5342 raft_consensus.cc:729] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47d50bedca44c14bd2feb55080c6436, State: Initialized, Role: FOLLOWER
I20260927 22:01:01.486667 5344 raft_consensus.cc:729] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f3b419e0044f16b72d697363b9c1ea, State: Initialized, Role: FOLLOWER
I20260927 22:01:01.486991 5342 consensus_queue.cc:260] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.487635 5342 ts_tablet_manager.cc:1428] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 22:01:01.487094 5344 consensus_queue.cc:260] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.489992 5344 ts_tablet_manager.cc:1428] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260927 22:01:01.495205 5368 ts_tablet_manager.cc:1918] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:01.495440 5368 log.cc:1196] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/5c5b3e36ff7b426baf90f38d27d69751
I20260927 22:01:01.496027 5368 ts_tablet_manager.cc:1939] T 5c5b3e36ff7b426baf90f38d27d69751 P a47d50bedca44c14bd2feb55080c6436: Deleting consensus metadata
I20260927 22:01:01.498056 4822 catalog_manager.cc:4841] TS a47d50bedca44c14bd2feb55080c6436 (127.3.159.1:34029): tablet 5c5b3e36ff7b426baf90f38d27d69751 (table test-table [id=23ba4fb8f9f8495d988c52d819e2ad37]) successfully deleted
I20260927 22:01:01.501844 5343 tablet_bootstrap.cc:492] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5: No bootstrap required, opened a new log
I20260927 22:01:01.502084 5343 ts_tablet_manager.cc:1397] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.011s
I20260927 22:01:01.502997 5343 raft_consensus.cc:348] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.503340 5343 raft_consensus.cc:374] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:01.503455 5343 raft_consensus.cc:729] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbfb3f083be48e99a813e3939b324a5, State: Initialized, Role: FOLLOWER
I20260927 22:01:01.503800 5343 consensus_queue.cc:260] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.505919 5343 ts_tablet_manager.cc:1428] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 22:01:01.667881 5350 raft_consensus.cc:482] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:01.668203 5350 raft_consensus.cc:504] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.670498 5350 leader_election.cc:288] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a47d50bedca44c14bd2feb55080c6436 (127.3.159.1:34029), 80f3b419e0044f16b72d697363b9c1ea (127.3.159.3:44003)
I20260927 22:01:01.679467 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.671429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:58265 (local address 127.3.159.3:44003)
0927 22:01:01.672029 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.672037 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.672672 (+ 635us) server_negotiation.cc:407] Connection header received
0927 22:01:01.672997 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.673003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.673098 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.673165 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:01.674128 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.675006 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.676997 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.678166 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.678312 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.678441 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.679027 (+ 586us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.679177 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423}
I20260927 22:01:01.680620 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.671010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:44003 (local address 127.3.159.2:58265)
0927 22:01:01.672510 (+ 1500us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:01.672533 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:01.672810 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:01.673426 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:01.673431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:01.673459 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:01.673914 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.673926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.675157 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.675171 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.676815 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.676830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.678598 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.678605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.678821 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.678858 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:01:01.680006 (+ 1148us) client_negotiation.cc:240] Negotiation successful
0927 22:01:01.680198 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"thread_start_us":1179,"threads_started":1}
I20260927 22:01:01.680569 5353 raft_consensus.cc:482] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:01.680912 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.672096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:34739 (local address 127.3.159.1:34029)
0927 22:01:01.672342 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.672350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.672381 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:01:01.672661 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.672668 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.672743 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.672864 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:01.673443 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.674489 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.677316 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.679107 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.679266 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.679373 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.680217 (+ 844us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.680455 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:01:01.680842 5353 raft_consensus.cc:504] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.681718 5353 leader_election.cc:288] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bbfb3f083be48e99a813e3939b324a5 (127.3.159.2:33313), 80f3b419e0044f16b72d697363b9c1ea (127.3.159.3:44003)
I20260927 22:01:01.681842 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.670933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:34029 (local address 127.3.159.2:34739)
0927 22:01:01.672164 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:01.672195 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:01.672382 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:01.672883 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:01.672887 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:01.672914 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:01.673239 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.673249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.674662 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.674667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.677106 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.677123 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.679495 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.679502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.679700 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.679740 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:01:01.681188 (+ 1448us) client_negotiation.cc:240] Negotiation successful
0927 22:01:01.681366 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"mutex_wait_us":1,"thread_start_us":864,"threads_started":1}
I20260927 22:01:01.682029 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9c82df14304d28967184dd1f59fc7b" candidate_uuid: "4bbfb3f083be48e99a813e3939b324a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f3b419e0044f16b72d697363b9c1ea" is_pre_election: true
I20260927 22:01:01.682374 5273 raft_consensus.cc:2454] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbfb3f083be48e99a813e3939b324a5 in term 0.
I20260927 22:01:01.682864 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9c82df14304d28967184dd1f59fc7b" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bbfb3f083be48e99a813e3939b324a5" is_pre_election: true
I20260927 22:01:01.682891 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9c82df14304d28967184dd1f59fc7b" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f3b419e0044f16b72d697363b9c1ea" is_pre_election: true
I20260927 22:01:01.682893 4993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9c82df14304d28967184dd1f59fc7b" candidate_uuid: "4bbfb3f083be48e99a813e3939b324a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47d50bedca44c14bd2feb55080c6436" is_pre_election: true
I20260927 22:01:01.683194 5273 raft_consensus.cc:2454] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47d50bedca44c14bd2feb55080c6436 in term 0.
I20260927 22:01:01.683202 5133 raft_consensus.cc:2454] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47d50bedca44c14bd2feb55080c6436 in term 0.
I20260927 22:01:01.683440 4993 raft_consensus.cc:2454] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bbfb3f083be48e99a813e3939b324a5 in term 0.
I20260927 22:01:01.683843 5064 leader_election.cc:302] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbfb3f083be48e99a813e3939b324a5, 80f3b419e0044f16b72d697363b9c1ea; no voters:
I20260927 22:01:01.684257 4925 leader_election.cc:302] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbfb3f083be48e99a813e3939b324a5, a47d50bedca44c14bd2feb55080c6436; no voters:
I20260927 22:01:01.684623 5350 raft_consensus.cc:2781] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:01.684708 5353 raft_consensus.cc:2781] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:01.684783 5350 raft_consensus.cc:482] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:01.684823 5353 raft_consensus.cc:482] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:01.685025 5350 raft_consensus.cc:3037] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:01.685035 5353 raft_consensus.cc:3037] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:01.689373 5350 raft_consensus.cc:504] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.689488 5353 raft_consensus.cc:504] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.690337 5350 leader_election.cc:288] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [CANDIDATE]: Term 1 election: Requested vote from peers a47d50bedca44c14bd2feb55080c6436 (127.3.159.1:34029), 80f3b419e0044f16b72d697363b9c1ea (127.3.159.3:44003)
I20260927 22:01:01.691015 4993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9c82df14304d28967184dd1f59fc7b" candidate_uuid: "4bbfb3f083be48e99a813e3939b324a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47d50bedca44c14bd2feb55080c6436"
I20260927 22:01:01.691381 4993 raft_consensus.cc:2379] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bbfb3f083be48e99a813e3939b324a5 in current term 1: Already voted for candidate a47d50bedca44c14bd2feb55080c6436 in this term.
I20260927 22:01:01.691346 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9c82df14304d28967184dd1f59fc7b" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bbfb3f083be48e99a813e3939b324a5"
I20260927 22:01:01.691640 5133 raft_consensus.cc:2379] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a47d50bedca44c14bd2feb55080c6436 in current term 1: Already voted for candidate 4bbfb3f083be48e99a813e3939b324a5 in this term.
I20260927 22:01:01.691569 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9c82df14304d28967184dd1f59fc7b" candidate_uuid: "4bbfb3f083be48e99a813e3939b324a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f3b419e0044f16b72d697363b9c1ea"
I20260927 22:01:01.691884 5272 raft_consensus.cc:3037] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:01.692073 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a9c82df14304d28967184dd1f59fc7b" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f3b419e0044f16b72d697363b9c1ea"
I20260927 22:01:01.692730 5353 leader_election.cc:288] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 election: Requested vote from peers 4bbfb3f083be48e99a813e3939b324a5 (127.3.159.2:33313), 80f3b419e0044f16b72d697363b9c1ea (127.3.159.3:44003)
I20260927 22:01:01.692780 4924 leader_election.cc:302] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a47d50bedca44c14bd2feb55080c6436; no voters: 4bbfb3f083be48e99a813e3939b324a5, 80f3b419e0044f16b72d697363b9c1ea
I20260927 22:01:01.693210 5353 raft_consensus.cc:2726] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 22:01:01.695287 5272 raft_consensus.cc:2454] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bbfb3f083be48e99a813e3939b324a5 in term 1.
I20260927 22:01:01.696002 5064 leader_election.cc:302] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bbfb3f083be48e99a813e3939b324a5, 80f3b419e0044f16b72d697363b9c1ea; no voters: a47d50bedca44c14bd2feb55080c6436
I20260927 22:01:01.696609 5350 raft_consensus.cc:2781] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:01.698825 5350 raft_consensus.cc:686] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 1 LEADER]: Becoming Leader. State: Replica: 4bbfb3f083be48e99a813e3939b324a5, State: Running, Role: LEADER
I20260927 22:01:01.699421 5350 consensus_queue.cc:237] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
W20260927 22:01:01.702042 5184 tablet.cc:2369] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:01.704169 4836 catalog_manager.cc:5495] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bbfb3f083be48e99a813e3939b324a5 (127.3.159.2). New cstate: current_term: 1 leader_uuid: "4bbfb3f083be48e99a813e3939b324a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:01:01 dist-test-slave-sg4j krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546458, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:01.785521 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.764853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44785 (local address 127.0.0.1:54336)
0927 22:01:01.766075 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:01.766094 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:01.766328 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:01.766721 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:01.766726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:01.767635 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:01.767864 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.767876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.769920 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.769927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.771032 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:01.771044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.771878 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.771884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:01.772096 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.772959 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:01.772978 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:01.776345 (+ 3367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:01.781631 (+ 5286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:01.781653 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:01.781663 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:01.782421 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:01.783035 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:01.783041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:01.783046 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:01.783274 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:01.784088 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:01.784100 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:01.784351 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 22:01:01.784592 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 22:01:01.784993 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"thread_start_us":1014,"threads_started":1}
I20260927 22:01:01.785665 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:01.764910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.3.159.62:44785)
0927 22:01:01.765718 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:01.765722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:01.766225 (+ 503us) server_negotiation.cc:407] Connection header received
0927 22:01:01.766474 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:01.766480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:01.766553 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:01.766684 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:01.768048 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.769753 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.771188 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:01.771680 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:01.771847 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:01.776730 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:01.776740 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:01.776746 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:01.776808 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:01.781348 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:01.782620 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:01.782628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:01.782632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:01.782835 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:01.783436 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:01.783445 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:01.783450 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:01.783889 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:01.784058 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:01.784692 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 22:01:01.785172 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":190,"server-negotiator.queue_time_us":677,"thread_start_us":503,"threads_started":1}
I20260927 22:01:01.793769 4836 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:54336:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:01:01.795369 4836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:01.805107 4973 tablet_service.cc:1467] Processing CreateTablet for tablet 973a94a15af24c54bc2244668f179aa9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=145e1411d66642018087c83a4205ec83]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:01:01.805743 5113 tablet_service.cc:1467] Processing CreateTablet for tablet 973a94a15af24c54bc2244668f179aa9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=145e1411d66642018087c83a4205ec83]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:01:01.805979 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973a94a15af24c54bc2244668f179aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:01.806458 5113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973a94a15af24c54bc2244668f179aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:01.806229 5253 tablet_service.cc:1467] Processing CreateTablet for tablet 973a94a15af24c54bc2244668f179aa9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=145e1411d66642018087c83a4205ec83]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:01:01.806979 5253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973a94a15af24c54bc2244668f179aa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:01.815528 5344 tablet_bootstrap.cc:492] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea: Bootstrap starting.
I20260927 22:01:01.817335 5342 tablet_bootstrap.cc:492] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: Bootstrap starting.
I20260927 22:01:01.818550 5344 tablet_bootstrap.cc:654] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:01.819408 5343 tablet_bootstrap.cc:492] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5: Bootstrap starting.
I20260927 22:01:01.819455 5342 tablet_bootstrap.cc:654] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:01.821574 5343 tablet_bootstrap.cc:654] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:01.831167 5342 tablet_bootstrap.cc:492] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: No bootstrap required, opened a new log
I20260927 22:01:01.831509 5342 ts_tablet_manager.cc:1397] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20260927 22:01:01.832459 5342 raft_consensus.cc:348] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.832908 5342 raft_consensus.cc:374] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:01.833069 5342 raft_consensus.cc:729] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47d50bedca44c14bd2feb55080c6436, State: Initialized, Role: FOLLOWER
I20260927 22:01:01.833487 5342 consensus_queue.cc:260] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.833607 5343 tablet_bootstrap.cc:492] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5: No bootstrap required, opened a new log
I20260927 22:01:01.833902 5343 ts_tablet_manager.cc:1397] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.012s
I20260927 22:01:01.834120 5353 tablet_replica.cc:439] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } }
I20260927 22:01:01.834412 5353 tablet_replica.cc:442] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:01.835047 5343 raft_consensus.cc:348] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.835378 5343 raft_consensus.cc:374] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:01.835453 5343 raft_consensus.cc:729] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbfb3f083be48e99a813e3939b324a5, State: Initialized, Role: FOLLOWER
I20260927 22:01:01.835911 5343 consensus_queue.cc:260] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.836524 5350 tablet_replica.cc:439] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } }
I20260927 22:01:01.836936 5350 tablet_replica.cc:442] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:01.839015 5342 ts_tablet_manager.cc:1428] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260927 22:01:01.840549 5343 ts_tablet_manager.cc:1428] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260927 22:01:01.844063 5344 tablet_bootstrap.cc:492] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea: No bootstrap required, opened a new log
I20260927 22:01:01.844295 5344 ts_tablet_manager.cc:1397] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.009s
I20260927 22:01:01.845196 5344 raft_consensus.cc:348] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.845511 5344 raft_consensus.cc:374] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:01.845598 5344 raft_consensus.cc:729] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f3b419e0044f16b72d697363b9c1ea, State: Initialized, Role: FOLLOWER
I20260927 22:01:01.845885 5344 consensus_queue.cc:260] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:01.846582 5344 ts_tablet_manager.cc:1428] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 22:01:01.846524 5348 tablet_replica.cc:439] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } }
I20260927 22:01:01.847028 5348 tablet_replica.cc:442] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:02.010460 5353 raft_consensus.cc:482] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:02.010677 5353 raft_consensus.cc:504] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:02.011445 5353 leader_election.cc:288] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bbfb3f083be48e99a813e3939b324a5 (127.3.159.2:33313), 80f3b419e0044f16b72d697363b9c1ea (127.3.159.3:44003)
I20260927 22:01:02.012081 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973a94a15af24c54bc2244668f179aa9" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f3b419e0044f16b72d697363b9c1ea" is_pre_election: true
I20260927 22:01:02.012373 5272 raft_consensus.cc:2454] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47d50bedca44c14bd2feb55080c6436 in term 0.
I20260927 22:01:02.012238 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973a94a15af24c54bc2244668f179aa9" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bbfb3f083be48e99a813e3939b324a5" is_pre_election: true
I20260927 22:01:02.012590 5133 raft_consensus.cc:2454] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47d50bedca44c14bd2feb55080c6436 in term 0.
I20260927 22:01:02.012969 4924 leader_election.cc:302] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80f3b419e0044f16b72d697363b9c1ea, a47d50bedca44c14bd2feb55080c6436; no voters:
I20260927 22:01:02.013334 5353 raft_consensus.cc:2781] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:02.013513 5353 raft_consensus.cc:482] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:02.013605 5353 raft_consensus.cc:3037] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:02.016034 5353 raft_consensus.cc:504] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:02.016769 5353 leader_election.cc:288] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 election: Requested vote from peers 4bbfb3f083be48e99a813e3939b324a5 (127.3.159.2:33313), 80f3b419e0044f16b72d697363b9c1ea (127.3.159.3:44003)
I20260927 22:01:02.017513 5133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973a94a15af24c54bc2244668f179aa9" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bbfb3f083be48e99a813e3939b324a5"
I20260927 22:01:02.017639 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973a94a15af24c54bc2244668f179aa9" candidate_uuid: "a47d50bedca44c14bd2feb55080c6436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f3b419e0044f16b72d697363b9c1ea"
I20260927 22:01:02.017819 5133 raft_consensus.cc:3037] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:02.017963 5272 raft_consensus.cc:3037] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:02.020280 5133 raft_consensus.cc:2454] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47d50bedca44c14bd2feb55080c6436 in term 1.
I20260927 22:01:02.020481 5272 raft_consensus.cc:2454] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47d50bedca44c14bd2feb55080c6436 in term 1.
I20260927 22:01:02.020953 4925 leader_election.cc:302] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bbfb3f083be48e99a813e3939b324a5, a47d50bedca44c14bd2feb55080c6436; no voters:
I20260927 22:01:02.021450 5353 raft_consensus.cc:2781] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:02.021836 5353 raft_consensus.cc:686] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [term 1 LEADER]: Becoming Leader. State: Replica: a47d50bedca44c14bd2feb55080c6436, State: Running, Role: LEADER
I20260927 22:01:02.022215 5353 consensus_queue.cc:237] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } }
I20260927 22:01:02.024423 5386 tablet_replica.cc:439] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: TxnStatusTablet state changed. Reason: New leader a47d50bedca44c14bd2feb55080c6436. Latest consensus state: current_term: 1 leader_uuid: "a47d50bedca44c14bd2feb55080c6436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } }
I20260927 22:01:02.024973 5386 tablet_replica.cc:442] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:01:02.025707 5388 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:01:02.025978 5388 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:01:02.026265 4836 catalog_manager.cc:5495] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 reported cstate change: term changed from 0 to 1, leader changed from <none> to a47d50bedca44c14bd2feb55080c6436 (127.3.159.1). New cstate: current_term: 1 leader_uuid: "a47d50bedca44c14bd2feb55080c6436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } health_report { overall_health: UNKNOWN } } }
I20260927 22:01:02.073129 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.066728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:34029 (local address 127.0.0.1:39698)
0927 22:01:02.066987 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:02.067012 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:02.067158 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:02.067761 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:02.067765 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:02.067776 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:01:02.067949 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:02.067957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.069619 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.069624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:02.070989 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:02.071005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.071840 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.071846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:02.072039 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.072107 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:01:02.072553 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:01:02.072580 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 22:01:02.072669 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 22:01:02.072825 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 22:01:02.073124 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.066813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.3.159.1:34029)
0927 22:01:02.067090 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:02.067097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:02.067128 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:01:02.067389 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:02.067396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:02.067537 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:02.067694 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:02.068120 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.069351 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.071177 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.071666 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.071824 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.072315 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:02.072436 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:02.072540 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:02.072693 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 22:01:02.072784 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 22:01:02.078418 5272 raft_consensus.cc:1261] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea [term 1 FOLLOWER]: Refusing update from remote peer a47d50bedca44c14bd2feb55080c6436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:02.079339 5386 consensus_queue.cc:1035] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:02.079761 5133 raft_consensus.cc:1261] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5 [term 1 FOLLOWER]: Refusing update from remote peer a47d50bedca44c14bd2feb55080c6436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:02.080653 5386 consensus_queue.cc:1035] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:02.084578 5350 tablet_replica.cc:439] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5: TxnStatusTablet state changed. Reason: New leader a47d50bedca44c14bd2feb55080c6436. Latest consensus state: current_term: 1 leader_uuid: "a47d50bedca44c14bd2feb55080c6436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } }
I20260927 22:01:02.085008 5350 tablet_replica.cc:442] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:02.087950 5393 mvcc.cc:205] Tried to move back new op lower bound from 7334078308664500224 to 7334078308446986240. Current Snapshot: MvccSnapshot[applied={T|T < 7334078308664500224}]
I20260927 22:01:02.091823 5390 mvcc.cc:205] Tried to move back new op lower bound from 7334078308664500224 to 7334078308446986240. Current Snapshot: MvccSnapshot[applied={T|T < 7334078308664500224}]
I20260927 22:01:02.097594 5350 tablet_replica.cc:439] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a47d50bedca44c14bd2feb55080c6436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } }
I20260927 22:01:02.098124 5350 tablet_replica.cc:442] T 973a94a15af24c54bc2244668f179aa9 P 4bbfb3f083be48e99a813e3939b324a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:02.098245 5348 tablet_replica.cc:439] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea: TxnStatusTablet state changed. Reason: New leader a47d50bedca44c14bd2feb55080c6436. Latest consensus state: current_term: 1 leader_uuid: "a47d50bedca44c14bd2feb55080c6436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } }
I20260927 22:01:02.098562 5348 tablet_replica.cc:442] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:02.101159 5391 mvcc.cc:205] Tried to move back new op lower bound from 7334078308664500224 to 7334078308446986240. Current Snapshot: MvccSnapshot[applied={T|T < 7334078308664500224}]
I20260927 22:01:02.103113 5386 tablet_replica.cc:439] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a47d50bedca44c14bd2feb55080c6436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } }
I20260927 22:01:02.103408 5386 tablet_replica.cc:442] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:01:02.108803 5348 tablet_replica.cc:439] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a47d50bedca44c14bd2feb55080c6436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } }
I20260927 22:01:02.109081 5348 tablet_replica.cc:442] T 973a94a15af24c54bc2244668f179aa9 P 80f3b419e0044f16b72d697363b9c1ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:02.121574 5350 consensus_queue.cc:1035] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:02.129000 5353 tablet_replica.cc:439] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a47d50bedca44c14bd2feb55080c6436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a47d50bedca44c14bd2feb55080c6436" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34029 } } peers { permanent_uuid: "4bbfb3f083be48e99a813e3939b324a5" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33313 } } peers { permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 } } }
I20260927 22:01:02.129570 5353 tablet_replica.cc:442] T 973a94a15af24c54bc2244668f179aa9 P a47d50bedca44c14bd2feb55080c6436: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:01:02.138758 5350 consensus_queue.cc:1035] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f3b419e0044f16b72d697363b9c1ea" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:02.146476 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.132447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35688 (local address 127.3.159.2:33313)
0927 22:01:02.135129 (+ 2682us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:02.135136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:02.136027 (+ 891us) server_negotiation.cc:407] Connection header received
0927 22:01:02.139350 (+ 3323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:02.139358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:02.139554 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:02.139740 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:02.140536 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.141361 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.142946 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.143679 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.144409 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.144593 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:02.145661 (+ 1068us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:02.145829 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:02.145948 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 22:01:02.146043 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2505,"thread_start_us":2309,"threads_started":1}
I20260927 22:01:02.170078 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.161431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:38091 (local address 127.3.159.1:34029)
0927 22:01:02.161679 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:02.161688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:02.162720 (+ 1032us) server_negotiation.cc:407] Connection header received
0927 22:01:02.162979 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:02.162984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:02.163085 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:02.163218 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:02.163863 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.164790 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.167678 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.168280 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.168429 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.169084 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:02.169316 (+ 232us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:02.169506 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:02.169679 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 22:01:02.169771 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:01:02.170272 5406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.161469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:34029 (local address 127.3.159.2:38091)
0927 22:01:02.162488 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:02.162517 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:02.162775 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:02.163281 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:02.163286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:02.163305 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:01:02.163660 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:02.163673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.164976 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.164982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:02.167218 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:02.167413 (+ 195us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.168541 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.168549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:02.168762 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.168856 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:01:02.169527 (+ 671us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:01:02.169540 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 22:01:02.169643 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 22:01:02.169828 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":482,"threads_started":1}
I20260927 22:01:02.237826 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.229446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:56657 (local address 127.3.159.2:33313)
0927 22:01:02.229714 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:02.229720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:02.229909 (+ 189us) server_negotiation.cc:407] Connection header received
0927 22:01:02.230186 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:02.230193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:02.230270 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:02.230372 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:02.231037 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.231921 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.235180 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.235936 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.236103 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.236935 (+ 832us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:02.237111 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:02.237180 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:02.237505 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 22:01:02.237592 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:01:02.238126 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.228747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:33313 (local address 127.3.159.1:56657)
0927 22:01:02.229774 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:02.229794 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:02.230029 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:02.230561 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:02.230566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:02.230583 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:01:02.230885 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:02.230894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.232091 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.232099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:02.234975 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:02.234997 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.236286 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.236294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:02.236539 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.236640 (+ 101us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:01:02.237310 (+ 670us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:01:02.237326 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 22:01:02.237447 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 22:01:02.237628 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":546,"threads_started":1}
I20260927 22:01:02.292501 4835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:02.292905 4835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:02.297425 4835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 227e13530a8646b7a410c2c3188fabef: Sending DeleteTablet for 3 replicas of tablet 7a9c82df14304d28967184dd1f59fc7b
I20260927 22:01:02.298679 5113 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9c82df14304d28967184dd1f59fc7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:02 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:35678
I20260927 22:01:02.299041 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9c82df14304d28967184dd1f59fc7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:02 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:39692
I20260927 22:01:02.299525 5253 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a9c82df14304d28967184dd1f59fc7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:02 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:37154
I20260927 22:01:02.335146 5416 tablet_replica.cc:330] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea: stopping tablet replica
I20260927 22:01:02.337244 5416 raft_consensus.cc:2229] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:02.337359 5415 tablet_replica.cc:330] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436: stopping tablet replica
I20260927 22:01:02.337707 5415 raft_consensus.cc:2229] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:02.337813 5416 raft_consensus.cc:2258] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:02.338124 5415 raft_consensus.cc:2258] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:02.339751 5416 ts_tablet_manager.cc:1905] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:02.345115 5414 tablet_replica.cc:330] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5: stopping tablet replica
I20260927 22:01:02.345584 5414 raft_consensus.cc:2229] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 1 LEADER]: Raft consensus shutting down.
Sep 27 22:01:02 dist-test-slave-sg4j krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546461, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:01:02.346330 5414 raft_consensus.cc:2258] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:02.346526 5415 ts_tablet_manager.cc:1905] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:02.351588 5414 ts_tablet_manager.cc:1905] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:02.360426 5416 ts_tablet_manager.cc:1918] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:01:02.360634 5416 log.cc:1196] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/7a9c82df14304d28967184dd1f59fc7b
I20260927 22:01:02.361145 5416 ts_tablet_manager.cc:1939] T 7a9c82df14304d28967184dd1f59fc7b P 80f3b419e0044f16b72d697363b9c1ea: Deleting consensus metadata
I20260927 22:01:02.367410 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.311772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.3.159.1:34029)
0927 22:01:02.312023 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:02.312031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:02.321712 (+ 9681us) server_negotiation.cc:407] Connection header received
0927 22:01:02.322384 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:02.322392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:02.322503 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:02.322598 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:02.329550 (+ 6952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.330884 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.335190 (+ 4306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.336317 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.336563 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.349735 (+ 13172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:02.349769 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:02.349795 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:02.349867 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:02.355760 (+ 5893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:02.363621 (+ 7861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:02.363637 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:02.363641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:02.363915 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:02.364491 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:02.364496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:02.364499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:02.365110 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:02.365522 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:02.366553 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0927 22:01:02.366948 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 22:01:02.370242 5038 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39710
I20260927 22:01:02.394366 5415 ts_tablet_manager.cc:1918] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:01:02.394608 5415 log.cc:1196] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/7a9c82df14304d28967184dd1f59fc7b
I20260927 22:01:02.395129 5415 ts_tablet_manager.cc:1939] T 7a9c82df14304d28967184dd1f59fc7b P a47d50bedca44c14bd2feb55080c6436: Deleting consensus metadata
I20260927 22:01:02.401228 4821 catalog_manager.cc:4841] TS 80f3b419e0044f16b72d697363b9c1ea (127.3.159.3:44003): tablet 7a9c82df14304d28967184dd1f59fc7b (table test-table [id=dbea8caada4547cdb7114c7046b7f425]) successfully deleted
I20260927 22:01:02.411203 5414 ts_tablet_manager.cc:1918] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:01:02.411458 5414 log.cc:1196] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/7a9c82df14304d28967184dd1f59fc7b
I20260927 22:01:02.412070 5414 ts_tablet_manager.cc:1939] T 7a9c82df14304d28967184dd1f59fc7b P 4bbfb3f083be48e99a813e3939b324a5: Deleting consensus metadata
I20260927 22:01:02.424130 5421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.379518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54346 (local address 127.3.159.62:44785)
0927 22:01:02.400398 (+ 20880us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:02.400408 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:02.400458 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:01:02.400549 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:02.400557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:02.400669 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:02.400857 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:02.403458 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.404820 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.407137 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.408010 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.408221 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.415004 (+ 6783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:02.415014 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:02.415019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:02.415081 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:02.420034 (+ 4953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:02.421150 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:02.421157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:02.421162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:02.421363 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:02.421897 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:02.421902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:02.421908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:02.422329 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:02.422647 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:02.423080 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 22:01:02.423403 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20685,"thread_start_us":21422,"threads_started":1}
W20260927 22:01:02.424978 4835 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54346
I20260927 22:01:02.428817 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 4907
I20260927 22:01:02.442445 4822 catalog_manager.cc:4841] TS a47d50bedca44c14bd2feb55080c6436 (127.3.159.1:34029): tablet 7a9c82df14304d28967184dd1f59fc7b (table test-table [id=dbea8caada4547cdb7114c7046b7f425]) successfully deleted
I20260927 22:01:02.453442 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.432395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.3.159.1:34029)
0927 22:01:02.432615 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:02.432621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:02.432683 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:01:02.432760 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:02.432766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:02.432881 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:02.433077 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:02.436032 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.437481 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.440431 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.441255 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.441463 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.445110 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:02.445118 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:02.445123 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:02.445187 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:02.449448 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:02.450573 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:02.450582 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:02.450587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:02.450925 (+ 338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:02.451402 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:02.451408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:02.451412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:02.451802 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:02.452065 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:02.452713 (+ 648us) server_negotiation.cc:299] Negotiation successful
0927 22:01:02.452992 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 22:01:02.455194 5038 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39716
W20260927 22:01:02.456166 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 4907: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:01:02.456429 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 4907
I20260927 22:01:02.469724 4822 catalog_manager.cc:4841] TS 4bbfb3f083be48e99a813e3939b324a5 (127.3.159.2:33313): tablet 7a9c82df14304d28967184dd1f59fc7b (table test-table [id=dbea8caada4547cdb7114c7046b7f425]) successfully deleted
I20260927 22:01:02.491565 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 5047
Sep 27 22:01:02 dist-test-slave-sg4j krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546461, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:01:02.515758 5402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.492777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.3.159.2:33313)
0927 22:01:02.493052 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:02.493060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:02.493085 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:01:02.493175 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:02.493182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:02.493299 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:02.493494 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:02.495053 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.496360 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.498316 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.499148 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.499310 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.505484 (+ 6174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:02.505510 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:02.505553 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:02.505697 (+ 144us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:02.510818 (+ 5121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:02.512329 (+ 1511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:02.512348 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:02.512353 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:02.512595 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:02.513350 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:02.513358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:02.513366 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:02.514019 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:02.514461 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:02.514955 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 22:01:02.515277 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 22:01:02.516782 5178 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35690
W20260927 22:01:02.517779 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 5047: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:01:02.517998 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 5047
I20260927 22:01:02.538762 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 5187
Sep 27 22:01:02 dist-test-slave-sg4j krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546461, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:01:02.559101 5423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:02.539637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37164 (local address 127.3.159.3:44003)
0927 22:01:02.540298 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:02.540303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:02.540326 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:01:02.540387 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:02.540390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:02.540467 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:02.540580 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:02.542200 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.542995 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.544706 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:02.545215 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:02.545358 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:02.551062 (+ 5704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:02.551082 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:02.551101 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:02.551149 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:02.555170 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:02.556394 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:02.556404 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:02.556406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:02.556619 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:02.557148 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:02.557152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:02.557155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:02.557698 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:02.557966 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:02.558433 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 22:01:02.558666 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":370,"threads_started":1}
W20260927 22:01:02.560039 5318 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37164
W20260927 22:01:02.560992 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 5187: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:01:02.561225 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 5187
I20260927 22:01:02.579180 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 4804
I20260927 22:01:02.580711 4876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:02.720429 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 4804
2026-09-27T22:01:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5655 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:01:04 dist-test-slave-sg4j krb5kdc[5427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:01:04 dist-test-slave-sg4j krb5kdc[5427](info): set up 2 sockets
Sep 27 22:01:04 dist-test-slave-sg4j krb5kdc[5427](info): commencing operation
krb5kdc: starting...
W20260927 22:01:05.763635 3708 mini_kdc.cc:121] Time spent starting KDC: real 2.992s user 0.005s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 22:01:05.958379 3708 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.195s user 0.000s sys 0.007s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:01:06 dist-test-slave-sg4j krb5kdc[5427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:01:06.150108 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38263
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:38263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:06.556844 5440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:06.557240 5440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:06.557451 5440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:06.566474 5440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:06.566623 5440 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:01:06.566668 5440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:06.566769 5440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:06.566852 5440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:06.566952 5440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:01:06.578614 5440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:38263
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:06.580380 5440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:06.582201 5440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:06.597031 5446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:06.597079 5448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:06.597066 5445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:06.598021 5440 server_base.cc:1034] running on GCE node
I20260927 22:01:06.598980 5440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:06.600869 5440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:06.602149 5440 hybrid_clock.cc:648] HybridClock initialized: now 1790546466602010 us; error 109 us; skew 500 ppm
I20260927 22:01:06.602623 5440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:06 dist-test-slave-sg4j krb5kdc[5427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546466, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:06.612178 5440 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:01:06.614519 5440 webserver.cc:460] Webserver started at http://127.3.159.62:33829/ using document root <none> and password file <none>
I20260927 22:01:06.615921 5440 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:06.616189 5440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:06.616586 5440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:06.620262 5440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "66a615240d8247708d6d8c539622a5a5"
format_stamp: "Formatted at 2026-09-27 22:01:06 on dist-test-slave-sg4j"
server_key: "5ed25457d4010c2a8f07e43e1a985813"
server_key_iv: "ac204c6e33c981c57534e1209f33ea83"
server_key_version: "encryptionkey@0"
I20260927 22:01:06.621236 5440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "66a615240d8247708d6d8c539622a5a5"
format_stamp: "Formatted at 2026-09-27 22:01:06 on dist-test-slave-sg4j"
server_key: "5ed25457d4010c2a8f07e43e1a985813"
server_key_iv: "ac204c6e33c981c57534e1209f33ea83"
server_key_version: "encryptionkey@0"
I20260927 22:01:06.628924 5440 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 22:01:06.634063 5455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:06.636185 5440 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:01:06.636452 5440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "66a615240d8247708d6d8c539622a5a5"
format_stamp: "Formatted at 2026-09-27 22:01:06 on dist-test-slave-sg4j"
server_key: "5ed25457d4010c2a8f07e43e1a985813"
server_key_iv: "ac204c6e33c981c57534e1209f33ea83"
server_key_version: "encryptionkey@0"
I20260927 22:01:06.636734 5440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:06.656116 5440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:06.661056 5440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:06.661361 5440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:06.687377 5440 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:38263
I20260927 22:01:06.687420 5516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:38263 every 8 connection(s)
I20260927 22:01:06.689553 5440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:06.695003 5517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:06.696591 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 5440
I20260927 22:01:06.696866 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:06.697873 3708 external_mini_cluster.cc:1428] Setting key 74f87e7dfe2b2600a52dce1430b27239
I20260927 22:01:06.714146 5517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5: Bootstrap starting.
Sep 27 22:01:06 dist-test-slave-sg4j krb5kdc[5427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:06.719591 5517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:06.721706 5517 log.cc:824] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:06.726760 5517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5: No bootstrap required, opened a new log
I20260927 22:01:06.729723 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:06.700670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60140 (local address 127.3.159.62:38263)
0927 22:01:06.701740 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:06.701753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:06.701815 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:01:06.703707 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:06.703737 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:06.704300 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:06.704741 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:06.707084 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:06.708549 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:06.710829 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:06.712027 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:06.712227 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:06.718262 (+ 6035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:06.718274 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:06.718295 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:06.718352 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:06.723647 (+ 5295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:06.724986 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:06.724995 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:06.725003 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:06.725257 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:06.725836 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:06.725840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:06.725843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:06.726636 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:06.726954 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:06.727727 (+ 773us) server_negotiation.cc:299] Negotiation successful
0927 22:01:06.728361 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":495,"threads_started":1}
I20260927 22:01:06.737777 5517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a615240d8247708d6d8c539622a5a5" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38263 } }
I20260927 22:01:06.738518 5517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:06.738694 5517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a615240d8247708d6d8c539622a5a5, State: Initialized, Role: FOLLOWER
I20260927 22:01:06.739852 5517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a615240d8247708d6d8c539622a5a5" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38263 } }
I20260927 22:01:06.740294 5517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:06.740473 5517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:06.740680 5517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:06.743557 5517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a615240d8247708d6d8c539622a5a5" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38263 } }
I20260927 22:01:06.744401 5517 leader_election.cc:302] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66a615240d8247708d6d8c539622a5a5; no voters:
I20260927 22:01:06.745177 5517 leader_election.cc:288] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:06.745453 5522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:06.747100 5522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [term 1 LEADER]: Becoming Leader. State: Replica: 66a615240d8247708d6d8c539622a5a5, State: Running, Role: LEADER
I20260927 22:01:06.747822 5522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a615240d8247708d6d8c539622a5a5" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38263 } }
I20260927 22:01:06.748199 5517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:06.751153 5524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66a615240d8247708d6d8c539622a5a5. Latest consensus state: current_term: 1 leader_uuid: "66a615240d8247708d6d8c539622a5a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a615240d8247708d6d8c539622a5a5" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38263 } } }
I20260927 22:01:06.751502 5523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66a615240d8247708d6d8c539622a5a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a615240d8247708d6d8c539622a5a5" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38263 } } }
I20260927 22:01:06.751688 5524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:06.751819 5523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:06.752789 5531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:06.761444 5531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:06.772166 5531 catalog_manager.cc:1347] Generated new cluster ID: f8b582393b8a4b2a8dd92d3d3fed06f3
I20260927 22:01:06.772317 5531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:06.788245 5531 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:06.789983 5531 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:06.804203 5531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5: Generated new TSK 0
I20260927 22:01:06.805850 5531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:01:06.875204 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38263
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:07.303344 5543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:07.303731 5543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:07.303917 5543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:07.312382 5543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:07.312541 5543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:07.312688 5543 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:01:07.312769 5543 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:01:07.312883 5543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:01:07.324901 5543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.159.62:38263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:07.327055 5543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:07.328898 5543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:07.346500 5548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:07.346617 5551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:07.346720 5549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:07.347698 5543 server_base.cc:1034] running on GCE node
I20260927 22:01:07.348587 5543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:07.350034 5543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:07.351367 5543 hybrid_clock.cc:648] HybridClock initialized: now 1790546467351246 us; error 59 us; skew 500 ppm
I20260927 22:01:07.351797 5543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:07 dist-test-slave-sg4j krb5kdc[5427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546467, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:07.356207 5543 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:01:07.358563 5543 webserver.cc:460] Webserver started at http://127.3.159.1:40091/ using document root <none> and password file <none>
I20260927 22:01:07.359527 5543 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:07.359697 5543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:07.360095 5543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:07.364120 5543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "8db739cd4d094f968c637e4a89b325ef"
format_stamp: "Formatted at 2026-09-27 22:01:07 on dist-test-slave-sg4j"
server_key: "44721af6d3c2b50e5c3690a78dbff53d"
server_key_iv: "4ea4673a01c7a7915e44a3ebb79d3195"
server_key_version: "encryptionkey@0"
I20260927 22:01:07.365322 5543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "8db739cd4d094f968c637e4a89b325ef"
format_stamp: "Formatted at 2026-09-27 22:01:07 on dist-test-slave-sg4j"
server_key: "44721af6d3c2b50e5c3690a78dbff53d"
server_key_iv: "4ea4673a01c7a7915e44a3ebb79d3195"
server_key_version: "encryptionkey@0"
I20260927 22:01:07.374042 5543 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260927 22:01:07.379522 5558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:07.381956 5543 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:01:07.382236 5543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "8db739cd4d094f968c637e4a89b325ef"
format_stamp: "Formatted at 2026-09-27 22:01:07 on dist-test-slave-sg4j"
server_key: "44721af6d3c2b50e5c3690a78dbff53d"
server_key_iv: "4ea4673a01c7a7915e44a3ebb79d3195"
server_key_version: "encryptionkey@0"
I20260927 22:01:07.382531 5543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:07.405597 5543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:07.411629 5543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:07.412009 5543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:07.415678 5543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:07.415840 5543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:07.415978 5543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:07.416075 5543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:01:07 dist-test-slave-sg4j krb5kdc[5427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546467, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:07.446386 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:07.416090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:41289 (local address 127.3.159.62:38263)
0927 22:01:07.417259 (+ 1169us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:07.417267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:07.418273 (+ 1006us) server_negotiation.cc:407] Connection header received
0927 22:01:07.420756 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:07.420762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:07.420873 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:07.420995 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:07.423597 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:07.424602 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:07.428190 (+ 3588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:07.429183 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:07.429547 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:07.435391 (+ 5844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:07.435405 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:07.435415 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:07.435499 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:07.440990 (+ 5491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:07.442592 (+ 1602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:07.442604 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:07.442611 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:07.442910 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:07.443530 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:07.443539 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:07.443546 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:07.444157 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:07.444385 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:07.445463 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0927 22:01:07.445855 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1004,"thread_start_us":873,"threads_started":1}
I20260927 22:01:07.447952 5568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:07.416913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38263 (local address 127.3.159.1:41289)
0927 22:01:07.418052 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:07.418108 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:07.420409 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:07.421283 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:07.421293 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:07.422285 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:07.423365 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:07.423394 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:07.424819 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:07.424823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:07.427869 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:07.427888 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:07.429656 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:07.429671 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:07.429935 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:07.430887 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:07.430941 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:07.434912 (+ 3971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:07.441298 (+ 6386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:07.441317 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:07.441347 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:07.442342 (+ 995us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:07.443096 (+ 754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:07.443099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:07.443102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:07.443377 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:07.444343 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:07.444358 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:07.444797 (+ 439us) client_negotiation.cc:769] Sending connection context
0927 22:01:07.445223 (+ 426us) client_negotiation.cc:240] Negotiation successful
0927 22:01:07.446037 (+ 814us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":466,"threads_started":1}
I20260927 22:01:07.478084 5543 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:44725
I20260927 22:01:07.478150 5678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:44725 every 8 connection(s)
I20260927 22:01:07.478932 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:07.456153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:50833 (local address 127.3.159.62:38263)
0927 22:01:07.456420 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:07.456427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:07.458437 (+ 2010us) server_negotiation.cc:407] Connection header received
0927 22:01:07.458737 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:07.458745 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:07.458887 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:07.459053 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:07.461136 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:07.462421 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:07.465466 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:07.466312 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:07.466497 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:07.469789 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:07.469801 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:07.469810 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:07.469896 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:07.474504 (+ 4608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:07.475829 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:07.475839 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:07.475845 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:07.476039 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:07.476685 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:07.476695 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:07.476701 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:07.477215 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:07.477461 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:07.478199 (+ 738us) server_negotiation.cc:299] Negotiation successful
0927 22:01:07.478492 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:01:07.478984 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:07.456635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38263 (local address 127.3.159.1:50833)
0927 22:01:07.458249 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:07.458277 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:07.458539 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:07.459124 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:07.459128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:07.460024 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:07.460816 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:07.460837 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:07.462650 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:07.462660 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:07.465094 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:07.465118 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:07.466802 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:07.466809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:07.466964 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:07.467826 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:07.467863 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:07.469444 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:07.474803 (+ 5359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:07.474818 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:07.474826 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:07.475647 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:07.476224 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:07.476234 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:07.476240 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:07.476520 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:07.477464 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:07.477471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:07.477802 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 22:01:07.478047 (+ 245us) client_negotiation.cc:240] Negotiation successful
0927 22:01:07.478549 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1361,"thread_start_us":1103,"threads_started":1}
I20260927 22:01:07.480082 5543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:01:07.483340 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 5543
I20260927 22:01:07.483624 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:01:07.484454 3708 external_mini_cluster.cc:1428] Setting key 6e5830dcf9e89f24761cba8da795df17
I20260927 22:01:07.485111 5679 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38263
I20260927 22:01:07.485461 5679 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:07.486263 5679 heartbeater.cc:511] Master 127.3.159.62:38263 requested a full tablet report, sending...
I20260927 22:01:07.489200 5472 ts_manager.cc:194] Registered new tserver with Master: 8db739cd4d094f968c637e4a89b325ef (127.3.159.1:44725)
I20260927 22:01:07.491021 5472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:41289
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:01:07.534339 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38263
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:07.901911 5683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:07.902259 5683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:07.902486 5683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:07.913178 5683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:07.913383 5683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:07.913575 5683 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:01:07.913666 5683 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:01:07.913877 5683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:01:07.925956 5683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.159.62:38263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:07.927690 5683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:07.929507 5683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:07.946859 5688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:07.947045 5691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:07.947058 5689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:07.949128 5683 server_base.cc:1034] running on GCE node
I20260927 22:01:07.950196 5683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:07.951601 5683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:07.952910 5683 hybrid_clock.cc:648] HybridClock initialized: now 1790546467952824 us; error 69 us; skew 500 ppm
I20260927 22:01:07.953528 5683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:07 dist-test-slave-sg4j krb5kdc[5427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546467, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:07.957566 5683 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:01:07.960156 5683 webserver.cc:460] Webserver started at http://127.3.159.2:37405/ using document root <none> and password file <none>
I20260927 22:01:07.961014 5683 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:07.961186 5683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:07.961608 5683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:07.965029 5683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "ad75e22fd120486ba70562d12a5d6780"
format_stamp: "Formatted at 2026-09-27 22:01:07 on dist-test-slave-sg4j"
server_key: "137cfe16e372f4b443dc7ef2f3f14322"
server_key_iv: "81c6de7d286d957887443b823c1eb8ce"
server_key_version: "encryptionkey@0"
I20260927 22:01:07.965983 5683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "ad75e22fd120486ba70562d12a5d6780"
format_stamp: "Formatted at 2026-09-27 22:01:07 on dist-test-slave-sg4j"
server_key: "137cfe16e372f4b443dc7ef2f3f14322"
server_key_iv: "81c6de7d286d957887443b823c1eb8ce"
server_key_version: "encryptionkey@0"
I20260927 22:01:07.973734 5683 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:01:07.978760 5698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:07.981163 5683 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:01:07.981666 5683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "ad75e22fd120486ba70562d12a5d6780"
format_stamp: "Formatted at 2026-09-27 22:01:07 on dist-test-slave-sg4j"
server_key: "137cfe16e372f4b443dc7ef2f3f14322"
server_key_iv: "81c6de7d286d957887443b823c1eb8ce"
server_key_version: "encryptionkey@0"
I20260927 22:01:07.981989 5683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:08.010109 5683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:08.015506 5683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:08.015867 5683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:08.019516 5683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:08.019673 5683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:08.019863 5683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:08.019981 5683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:01:08 dist-test-slave-sg4j krb5kdc[5427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546467, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:08.050189 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.020204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:49159 (local address 127.3.159.62:38263)
0927 22:01:08.023626 (+ 3422us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:08.023633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:08.023660 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:01:08.025241 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:08.025247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:08.025353 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:08.025498 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:08.028096 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.029342 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.031702 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.032315 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.032473 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.038824 (+ 6351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:08.038836 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:08.038843 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:08.038907 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:08.044498 (+ 5591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:08.046089 (+ 1591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:08.046099 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:08.046107 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:08.046362 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:08.047092 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:08.047111 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:08.047114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:08.047542 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:08.047735 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:08.049411 (+ 1676us) server_negotiation.cc:299] Negotiation successful
0927 22:01:08.049742 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3271,"thread_start_us":3146,"threads_started":1}
I20260927 22:01:08.053026 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.020947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38263 (local address 127.3.159.2:49159)
0927 22:01:08.022431 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:08.022486 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:08.024897 (+ 2411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:08.025786 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:08.025795 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:08.026675 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:08.027796 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.027829 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.029555 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.029561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.031466 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.031482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.032658 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.032666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.032939 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.034272 (+ 1333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:08.034307 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:08.038347 (+ 4040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:08.044845 (+ 6498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:08.044867 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:08.044893 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:08.045855 (+ 962us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:08.046553 (+ 698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:08.046561 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:08.046567 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:08.046916 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:08.047794 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:08.047809 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:08.048701 (+ 892us) client_negotiation.cc:769] Sending connection context
0927 22:01:08.049257 (+ 556us) client_negotiation.cc:240] Negotiation successful
0927 22:01:08.050321 (+ 1064us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1085,"thread_start_us":1108,"threads_started":1}
I20260927 22:01:08.080256 5767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.062123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38263 (local address 127.3.159.2:59183)
0927 22:01:08.062908 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:08.062936 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:08.063141 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:08.063808 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:08.063812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:08.064633 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:08.065128 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.065139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.067003 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.067011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.069161 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.069185 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.070371 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.070378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.070560 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.071711 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:08.071730 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:08.073079 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:08.077012 (+ 3933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:08.077020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:08.077025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:08.077676 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:08.078205 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:08.078209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:08.078211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:08.078336 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:08.078948 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:08.078953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:08.079198 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 22:01:08.079417 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 22:01:08.079842 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":485,"threads_started":1}
I20260927 22:01:08.080672 5707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.061682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:59183 (local address 127.3.159.62:38263)
0927 22:01:08.061970 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:08.061979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:08.063083 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 22:01:08.063371 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:08.063381 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:08.063542 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:08.063744 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:08.065462 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.066757 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.069439 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.070130 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.070273 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.073309 (+ 3036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:08.073314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:08.073317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:08.073350 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:08.076841 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:08.077897 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:08.077906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:08.077912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:08.078084 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:08.078461 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:08.078469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:08.078474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:08.078814 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:08.078967 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:08.079876 (+ 909us) server_negotiation.cc:299] Negotiation successful
0927 22:01:08.080183 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:01:08.087759 5683 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:34645
I20260927 22:01:08.087838 5818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:34645 every 8 connection(s)
I20260927 22:01:08.089641 5683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:01:08.092615 5819 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38263
I20260927 22:01:08.092980 5819 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:08.093818 5819 heartbeater.cc:511] Master 127.3.159.62:38263 requested a full tablet report, sending...
I20260927 22:01:08.095538 5472 ts_manager.cc:194] Registered new tserver with Master: ad75e22fd120486ba70562d12a5d6780 (127.3.159.2:34645)
I20260927 22:01:08.096405 5472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:49159
I20260927 22:01:08.099982 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 5683
I20260927 22:01:08.100244 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:01:08.100752 3708 external_mini_cluster.cc:1428] Setting key 3956d43cc958de9e69f654d8d9db6908
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:01:08.151021 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38263
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:08.495061 5679 heartbeater.cc:503] Master 127.3.159.62:38263 was elected leader, sending a full tablet report...
W20260927 22:01:08.559118 5823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:08.559603 5823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:08.559808 5823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:08.568619 5823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:08.568861 5823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:08.569037 5823 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 22:01:08.569130 5823 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:01:08.569273 5823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:01:08.581166 5823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.159.62:38263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:08.583081 5823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:08.584910 5823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:08.604331 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:08.604461 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:08.604709 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:08.605636 5823 server_base.cc:1034] running on GCE node
I20260927 22:01:08.606482 5823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:08.607759 5823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:08.609163 5823 hybrid_clock.cc:648] HybridClock initialized: now 1790546468609075 us; error 84 us; skew 500 ppm
I20260927 22:01:08.609650 5823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:08 dist-test-slave-sg4j krb5kdc[5427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:08.613530 5823 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:01:08.615583 5823 webserver.cc:460] Webserver started at http://127.3.159.3:43873/ using document root <none> and password file <none>
I20260927 22:01:08.616449 5823 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:08.616577 5823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:08.616907 5823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:08.621155 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "6ef955e7a5124be1b5161fd19d48f6f2"
format_stamp: "Formatted at 2026-09-27 22:01:08 on dist-test-slave-sg4j"
server_key: "7a8cd8fae85e4e70e0fa94c5ba7c070c"
server_key_iv: "17d7919003ea7bbc970bbca9129c33c6"
server_key_version: "encryptionkey@0"
I20260927 22:01:08.622138 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "6ef955e7a5124be1b5161fd19d48f6f2"
format_stamp: "Formatted at 2026-09-27 22:01:08 on dist-test-slave-sg4j"
server_key: "7a8cd8fae85e4e70e0fa94c5ba7c070c"
server_key_iv: "17d7919003ea7bbc970bbca9129c33c6"
server_key_version: "encryptionkey@0"
I20260927 22:01:08.629232 5823 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260927 22:01:08.634744 5838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:08.637054 5823 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:01:08.637367 5823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "6ef955e7a5124be1b5161fd19d48f6f2"
format_stamp: "Formatted at 2026-09-27 22:01:08 on dist-test-slave-sg4j"
server_key: "7a8cd8fae85e4e70e0fa94c5ba7c070c"
server_key_iv: "17d7919003ea7bbc970bbca9129c33c6"
server_key_version: "encryptionkey@0"
I20260927 22:01:08.637694 5823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:08.658938 5823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:08.665016 5823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:08.665516 5823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:08.669692 5823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:08.669865 5823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:08.670030 5823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:08.670130 5823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:01:08 dist-test-slave-sg4j krb5kdc[5427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:08.701007 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.670764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:40341 (local address 127.3.159.62:38263)
0927 22:01:08.671391 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:08.671398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:08.673775 (+ 2377us) server_negotiation.cc:407] Connection header received
0927 22:01:08.676924 (+ 3149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:08.676931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:08.677066 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:08.677250 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:08.680413 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.681922 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.684809 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.685768 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.686002 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.691376 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:08.691385 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:08.691393 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:08.691450 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:08.696801 (+ 5351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:08.698028 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:08.698036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:08.698042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:08.698370 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:08.698919 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:08.698924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:08.698928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:08.699259 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:08.699415 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:08.700246 (+ 831us) server_negotiation.cc:299] Negotiation successful
0927 22:01:08.700533 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":372,"threads_started":1}
I20260927 22:01:08.702679 5848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.672029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38263 (local address 127.3.159.3:40341)
0927 22:01:08.673369 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:08.673594 (+ 225us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:08.676332 (+ 2738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:08.677830 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:08.677841 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:08.678857 (+ 1016us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:08.679964 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.679993 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.682237 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.682243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.684543 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.684562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.686117 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.686123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.686441 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.687483 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:08.687511 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:08.690969 (+ 3458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:08.697092 (+ 6123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:08.697111 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:08.697138 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:08.697859 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:08.698551 (+ 692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:08.698556 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:08.698560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:08.698788 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:08.699406 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:08.699415 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:08.699835 (+ 420us) client_negotiation.cc:769] Sending connection context
0927 22:01:08.700149 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 22:01:08.700768 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":538,"threads_started":1}
I20260927 22:01:08.733305 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.711812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38263 (local address 127.3.159.3:56989)
0927 22:01:08.712772 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:08.712802 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:08.713099 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:08.713607 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:08.713614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:08.714586 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:08.715306 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.715331 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.716897 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.716908 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.719966 (+ 3058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.720001 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.721321 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.721334 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.721727 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.722710 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:08.722746 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:08.724356 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:08.729321 (+ 4965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:08.729330 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:08.729336 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:08.730237 (+ 901us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:08.730811 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:08.730817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:08.730821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:08.730984 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:08.731819 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:08.731828 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:08.732130 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 22:01:08.732322 (+ 192us) client_negotiation.cc:240] Negotiation successful
0927 22:01:08.732792 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":555,"threads_started":1}
I20260927 22:01:08.733901 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.712327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:56989 (local address 127.3.159.62:38263)
0927 22:01:08.712573 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:08.712582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:08.713003 (+ 421us) server_negotiation.cc:407] Connection header received
0927 22:01:08.713268 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:08.713272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:08.713351 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:08.713549 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:08.715588 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.716676 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.720366 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.720975 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.721187 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.724780 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:08.724788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:08.724792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:08.724853 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:08.729033 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:08.730452 (+ 1419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:08.730459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:08.730465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:08.730646 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:08.731151 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:08.731159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:08.731163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:08.731581 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:08.731746 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:08.733005 (+ 1259us) server_negotiation.cc:299] Negotiation successful
0927 22:01:08.733245 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:01:08.747010 5823 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:39939
I20260927 22:01:08.747066 5958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:39939 every 8 connection(s)
I20260927 22:01:08.748809 5823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:01:08.751822 5959 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38263
I20260927 22:01:08.752353 5959 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:08.753489 5959 heartbeater.cc:511] Master 127.3.159.62:38263 requested a full tablet report, sending...
I20260927 22:01:08.755585 5472 ts_manager.cc:194] Registered new tserver with Master: 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939)
I20260927 22:01:08.756680 5472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:40341
I20260927 22:01:08.757833 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 5823
I20260927 22:01:08.758065 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:01:08.758754 3708 external_mini_cluster.cc:1428] Setting key 50a6f2d0c274645acad0beef90562d26
I20260927 22:01:08.763242 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:01:08 dist-test-slave-sg4j krb5kdc[5427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:01:08 dist-test-slave-sg4j krb5kdc[5427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:08.823902 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.805272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60144 (local address 127.3.159.62:38263)
0927 22:01:08.805541 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:08.805546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:08.805941 (+ 395us) server_negotiation.cc:407] Connection header received
0927 22:01:08.806155 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:08.806159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:08.806286 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:08.806473 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:08.808775 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.809810 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.811413 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.811895 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.812020 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.816956 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:08.816963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:08.816967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:08.817015 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:08.820305 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:08.821458 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:08.821467 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:08.821471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:08.821650 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:08.822098 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:08.822102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:08.822104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:08.822507 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:08.822731 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:08.823354 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 22:01:08.823585 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260927 22:01:08.830075 5472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:01:08.838907 5472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:08.870246 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.856380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.3.159.3:39939)
0927 22:01:08.858666 (+ 2286us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:08.858676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:08.858716 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:01:08.858864 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:08.858873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:08.859208 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:08.860607 (+ 1399us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:08.861087 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.863032 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.866866 (+ 3834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.868572 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.868763 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.869100 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:08.869546 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 22:01:08.869795 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2066,"thread_start_us":2147,"threads_started":1}
I20260927 22:01:08.870397 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.856079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:39939 (local address 127.0.0.1:49592)
0927 22:01:08.857422 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:08.857503 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:08.857814 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:08.859643 (+ 1829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:08.859653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:08.859704 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:08.860387 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.860404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.863267 (+ 2863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.863274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.866580 (+ 3306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.866596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.868863 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.868872 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.869208 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.869263 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 22:01:08.869490 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 22:01:08.869795 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":5,"thread_start_us":8568,"threads_started":1}
I20260927 22:01:08.875073 5976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.856308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:34645 (local address 127.0.0.1:35384)
0927 22:01:08.864015 (+ 7707us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:08.864041 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:08.864229 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:08.865140 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:08.865146 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:08.865180 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:08.865787 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.865801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.867799 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.867808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.871515 (+ 3707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.871537 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.874087 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.874094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.874279 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.874324 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:01:08.874441 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 22:01:08.874617 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7527,"mutex_wait_us":44,"thread_start_us":878,"threads_started":1}
I20260927 22:01:08.875177 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.856936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.3.159.2:34645)
0927 22:01:08.860425 (+ 3489us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:08.860437 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:08.864234 (+ 3797us) server_negotiation.cc:407] Connection header received
0927 22:01:08.864464 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:08.864478 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:08.864843 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:08.865149 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:08.866018 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.867631 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.871789 (+ 4158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.873622 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.873794 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.874067 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:08.874550 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 22:01:08.874778 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3273,"thread_start_us":5565,"threads_started":1}
I20260927 22:01:08.876499 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.861006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.3.159.1:44725)
0927 22:01:08.863363 (+ 2357us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:08.863377 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:08.865792 (+ 2415us) server_negotiation.cc:407] Connection header received
0927 22:01:08.866169 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:08.866181 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:08.866580 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:08.866831 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:08.867931 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.869794 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.872581 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.874370 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.874567 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.874793 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:08.875958 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0927 22:01:08.876167 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2114,"thread_start_us":1913,"threads_started":1}
I20260927 22:01:08.877207 5972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.860271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:44725 (local address 127.0.0.1:36912)
0927 22:01:08.865630 (+ 5359us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:08.865657 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:08.865934 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:08.867184 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:08.867189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:08.867219 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:08.867644 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.867657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.870057 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.870063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.872331 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.872355 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.874993 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.875005 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.875677 (+ 672us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.875717 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:01:08.876688 (+ 971us) client_negotiation.cc:240] Negotiation successful
0927 22:01:08.876881 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5165,"thread_start_us":3324,"threads_started":1}
I20260927 22:01:08.875801 5893 tablet_service.cc:1467] Processing CreateTablet for tablet dc9d2a5963cd401c8cd736b31dd3f0a4 (DEFAULT_TABLE table=test-table [id=4b1f11fca0454db09cdb5962419e9620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:08.878543 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9d2a5963cd401c8cd736b31dd3f0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:08.879642 5749 tablet_service.cc:1467] Processing CreateTablet for tablet dc9d2a5963cd401c8cd736b31dd3f0a4 (DEFAULT_TABLE table=test-table [id=4b1f11fca0454db09cdb5962419e9620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:08.881156 5749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9d2a5963cd401c8cd736b31dd3f0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:08.881537 5609 tablet_service.cc:1467] Processing CreateTablet for tablet dc9d2a5963cd401c8cd736b31dd3f0a4 (DEFAULT_TABLE table=test-table [id=4b1f11fca0454db09cdb5962419e9620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:08.883448 5609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9d2a5963cd401c8cd736b31dd3f0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:08.897013 5978 tablet_bootstrap.cc:492] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780: Bootstrap starting.
I20260927 22:01:08.902303 5979 tablet_bootstrap.cc:492] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: Bootstrap starting.
I20260927 22:01:08.903378 5978 tablet_bootstrap.cc:654] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:08.905889 5978 log.cc:824] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:08.907493 5980 tablet_bootstrap.cc:492] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: Bootstrap starting.
I20260927 22:01:08.911674 5980 tablet_bootstrap.cc:654] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:08.913211 5980 log.cc:824] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:08.913777 5979 tablet_bootstrap.cc:654] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:08.915550 5979 log.cc:824] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:08.918761 5978 tablet_bootstrap.cc:492] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780: No bootstrap required, opened a new log
I20260927 22:01:08.919240 5978 ts_tablet_manager.cc:1397] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.003s
I20260927 22:01:08.920421 5979 tablet_bootstrap.cc:492] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: No bootstrap required, opened a new log
I20260927 22:01:08.920634 5980 tablet_bootstrap.cc:492] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: No bootstrap required, opened a new log
I20260927 22:01:08.920859 5979 ts_tablet_manager.cc:1397] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20260927 22:01:08.920971 5980 ts_tablet_manager.cc:1397] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260927 22:01:08.928234 5979 raft_consensus.cc:348] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } }
I20260927 22:01:08.929045 5979 raft_consensus.cc:374] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:08.929170 5979 raft_consensus.cc:729] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef955e7a5124be1b5161fd19d48f6f2, State: Initialized, Role: FOLLOWER
I20260927 22:01:08.930049 5979 consensus_queue.cc:260] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } }
I20260927 22:01:08.931352 5959 heartbeater.cc:503] Master 127.3.159.62:38263 was elected leader, sending a full tablet report...
I20260927 22:01:08.932156 5979 ts_tablet_manager.cc:1428] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260927 22:01:08.932541 5978 raft_consensus.cc:348] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } }
I20260927 22:01:08.933172 5978 raft_consensus.cc:374] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:08.933306 5978 raft_consensus.cc:729] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad75e22fd120486ba70562d12a5d6780, State: Initialized, Role: FOLLOWER
I20260927 22:01:08.933261 5980 raft_consensus.cc:348] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } }
I20260927 22:01:08.933760 5980 raft_consensus.cc:374] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:08.933863 5980 raft_consensus.cc:729] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db739cd4d094f968c637e4a89b325ef, State: Initialized, Role: FOLLOWER
I20260927 22:01:08.934182 5978 consensus_queue.cc:260] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } }
I20260927 22:01:08.934872 5980 consensus_queue.cc:260] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } }
I20260927 22:01:08.935423 5985 raft_consensus.cc:482] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:08.937317 5819 heartbeater.cc:503] Master 127.3.159.62:38263 was elected leader, sending a full tablet report...
I20260927 22:01:08.937868 5980 ts_tablet_manager.cc:1428] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: Time spent starting tablet: real 0.017s user 0.019s sys 0.000s
I20260927 22:01:08.937881 5978 ts_tablet_manager.cc:1428] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260927 22:01:08.935813 5985 raft_consensus.cc:504] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } }
I20260927 22:01:08.943940 5985 leader_election.cc:288] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939), ad75e22fd120486ba70562d12a5d6780 (127.3.159.2:34645)
I20260927 22:01:08.951877 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.942659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:37687 (local address 127.3.159.3:39939)
0927 22:01:08.942917 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:08.942925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:08.943663 (+ 738us) server_negotiation.cc:407] Connection header received
0927 22:01:08.943934 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:08.943940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:08.944016 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:08.944090 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:08.945753 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.947132 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.949214 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.950664 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.950783 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.950889 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:08.951373 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 22:01:08.951554 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:01:08.951920 5989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.942407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:39939 (local address 127.3.159.1:37687)
0927 22:01:08.943540 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:08.943559 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:08.943733 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:08.944301 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:08.944305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:08.944326 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:08.945542 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.945556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.947297 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.947302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.949021 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.949036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.950893 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.950900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.951095 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.951143 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:01:08.951296 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 22:01:08.951467 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"thread_start_us":921,"threads_started":1}
I20260927 22:01:08.952459 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.942634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:56481 (local address 127.3.159.2:34645)
0927 22:01:08.942841 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:08.942846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:08.945296 (+ 2450us) server_negotiation.cc:407] Connection header received
0927 22:01:08.945551 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:08.945557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:08.945640 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:08.945741 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:08.946460 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.947581 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.950177 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.951246 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.951392 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.951498 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:08.952061 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 22:01:08.952226 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:01:08.952695 5988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:08.942407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:34645 (local address 127.3.159.1:56481)
0927 22:01:08.945164 (+ 2757us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:08.945183 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:08.945330 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:08.945774 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:08.945779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:08.945809 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:08.946226 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.946240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.947734 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.947754 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.949940 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:08.949955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:08.951517 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:08.951524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:08.951841 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:08.951887 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:01:08.952011 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 22:01:08.952245 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2588,"mutex_wait_us":29,"thread_start_us":2353,"threads_started":1}
I20260927 22:01:08.953866 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9d2a5963cd401c8cd736b31dd3f0a4" candidate_uuid: "8db739cd4d094f968c637e4a89b325ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" is_pre_election: true
I20260927 22:01:08.954104 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9d2a5963cd401c8cd736b31dd3f0a4" candidate_uuid: "8db739cd4d094f968c637e4a89b325ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad75e22fd120486ba70562d12a5d6780" is_pre_election: true
I20260927 22:01:08.954587 5914 raft_consensus.cc:2454] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db739cd4d094f968c637e4a89b325ef in term 0.
I20260927 22:01:08.954738 5774 raft_consensus.cc:2454] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db739cd4d094f968c637e4a89b325ef in term 0.
I20260927 22:01:08.955514 5561 leader_election.cc:302] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef955e7a5124be1b5161fd19d48f6f2, 8db739cd4d094f968c637e4a89b325ef; no voters:
I20260927 22:01:08.956166 5985 raft_consensus.cc:2781] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:08.956499 5985 raft_consensus.cc:482] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:08.956681 5985 raft_consensus.cc:3037] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:08.959151 5985 raft_consensus.cc:504] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } }
I20260927 22:01:08.959899 5985 leader_election.cc:288] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [CANDIDATE]: Term 1 election: Requested vote from peers 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939), ad75e22fd120486ba70562d12a5d6780 (127.3.159.2:34645)
I20260927 22:01:08.960608 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9d2a5963cd401c8cd736b31dd3f0a4" candidate_uuid: "8db739cd4d094f968c637e4a89b325ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad75e22fd120486ba70562d12a5d6780"
I20260927 22:01:08.960851 5774 raft_consensus.cc:3037] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:08.960772 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9d2a5963cd401c8cd736b31dd3f0a4" candidate_uuid: "8db739cd4d094f968c637e4a89b325ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef955e7a5124be1b5161fd19d48f6f2"
I20260927 22:01:08.961194 5914 raft_consensus.cc:3037] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:08.963394 5774 raft_consensus.cc:2454] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db739cd4d094f968c637e4a89b325ef in term 1.
I20260927 22:01:08.964012 5559 leader_election.cc:302] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8db739cd4d094f968c637e4a89b325ef, ad75e22fd120486ba70562d12a5d6780; no voters:
I20260927 22:01:08.964404 5914 raft_consensus.cc:2454] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db739cd4d094f968c637e4a89b325ef in term 1.
I20260927 22:01:08.964483 5985 raft_consensus.cc:2781] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:08.964713 5985 raft_consensus.cc:686] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 1 LEADER]: Becoming Leader. State: Replica: 8db739cd4d094f968c637e4a89b325ef, State: Running, Role: LEADER
I20260927 22:01:08.965428 5985 consensus_queue.cc:237] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } }
I20260927 22:01:08.970717 5472 catalog_manager.cc:5495] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 8db739cd4d094f968c637e4a89b325ef (127.3.159.1). New cstate: current_term: 1 leader_uuid: "8db739cd4d094f968c637e4a89b325ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } health_report { overall_health: HEALTHY } } }
W20260927 22:01:08.983990 5680 tablet.cc:2369] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:01:09.001349 5960 tablet.cc:2369] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 22:01:09 dist-test-slave-sg4j krb5kdc[5427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546466, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:09.038233 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.021825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.3.159.62:38263)
0927 22:01:09.022188 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:09.022193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:09.022963 (+ 770us) server_negotiation.cc:407] Connection header received
0927 22:01:09.023199 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:09.023203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:09.023296 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:09.023405 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:09.024961 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.025917 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.027300 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.027742 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.027892 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.031897 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:09.031903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:09.031906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:09.031938 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:09.035064 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:09.035948 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:09.035956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:09.035961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:09.036099 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:09.036479 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:09.036487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:09.036490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:09.036841 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:09.037031 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:09.037503 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 22:01:09.037782 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260927 22:01:09.038362 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.021354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38263 (local address 127.0.0.1:60150)
0927 22:01:09.022615 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:09.022642 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:09.023009 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:09.023484 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:09.023488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:09.024439 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:09.024754 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.024764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.026053 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.026062 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.027110 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.027121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.027946 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.027953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.028132 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.028949 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:09.028967 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:09.031635 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:09.035286 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:09.035305 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:09.035315 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:09.035818 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:09.036209 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:09.036215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:09.036222 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:09.036373 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:09.036983 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:09.036999 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:09.037182 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 22:01:09.037372 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 22:01:09.037927 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":182,"thread_start_us":754,"threads_started":1}
I20260927 22:01:09.045679 5472 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:60150:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:01:09.047029 5472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:09.056293 5609 tablet_service.cc:1467] Processing CreateTablet for tablet 4353387fbe344609a6c235d221d6de86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b615ae36f8e24cb3a608d7d983b4b685]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:01:09.056552 5749 tablet_service.cc:1467] Processing CreateTablet for tablet 4353387fbe344609a6c235d221d6de86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b615ae36f8e24cb3a608d7d983b4b685]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:01:09.057499 5609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4353387fbe344609a6c235d221d6de86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:09.056854 5893 tablet_service.cc:1467] Processing CreateTablet for tablet 4353387fbe344609a6c235d221d6de86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b615ae36f8e24cb3a608d7d983b4b685]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:01:09.057796 5749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4353387fbe344609a6c235d221d6de86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:09.057852 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4353387fbe344609a6c235d221d6de86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:09.064404 5978 tablet_bootstrap.cc:492] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: Bootstrap starting.
I20260927 22:01:09.065521 5980 tablet_bootstrap.cc:492] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef: Bootstrap starting.
I20260927 22:01:09.065773 5979 tablet_bootstrap.cc:492] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2: Bootstrap starting.
I20260927 22:01:09.066493 5978 tablet_bootstrap.cc:654] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:09.067379 5979 tablet_bootstrap.cc:654] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:09.067378 5980 tablet_bootstrap.cc:654] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:09.070986 5978 tablet_bootstrap.cc:492] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: No bootstrap required, opened a new log
I20260927 22:01:09.071275 5978 ts_tablet_manager.cc:1397] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 22:01:09.071799 5980 tablet_bootstrap.cc:492] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef: No bootstrap required, opened a new log
I20260927 22:01:09.071987 5980 ts_tablet_manager.cc:1397] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260927 22:01:09.072331 5978 raft_consensus.cc:348] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.072628 5978 raft_consensus.cc:374] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:09.072741 5978 raft_consensus.cc:729] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad75e22fd120486ba70562d12a5d6780, State: Initialized, Role: FOLLOWER
I20260927 22:01:09.072922 5979 tablet_bootstrap.cc:492] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2: No bootstrap required, opened a new log
I20260927 22:01:09.073098 5979 ts_tablet_manager.cc:1397] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 22:01:09.073026 5978 consensus_queue.cc:260] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.072865 5980 raft_consensus.cc:348] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.073271 5980 raft_consensus.cc:374] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:09.073382 5980 raft_consensus.cc:729] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db739cd4d094f968c637e4a89b325ef, State: Initialized, Role: FOLLOWER
I20260927 22:01:09.073865 5978 ts_tablet_manager.cc:1428] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:01:09.073760 5986 tablet_replica.cc:439] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } }
I20260927 22:01:09.074221 5986 tablet_replica.cc:442] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:09.074055 5979 raft_consensus.cc:348] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.074332 5979 raft_consensus.cc:374] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:09.074438 5979 raft_consensus.cc:729] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef955e7a5124be1b5161fd19d48f6f2, State: Initialized, Role: FOLLOWER
I20260927 22:01:09.074354 5980 consensus_queue.cc:260] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.074716 5979 consensus_queue.cc:260] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.074966 5980 ts_tablet_manager.cc:1428] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 22:01:09.075397 5979 ts_tablet_manager.cc:1428] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 22:01:09.075296 5984 tablet_replica.cc:439] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } }
I20260927 22:01:09.075740 5984 tablet_replica.cc:442] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:09.077502 5985 tablet_replica.cc:439] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } }
I20260927 22:01:09.077997 5985 tablet_replica.cc:442] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 22:01:09.092541 5820 tablet.cc:2369] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:09.098778 5986 raft_consensus.cc:482] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:09.099063 5986 raft_consensus.cc:504] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.101207 5986 leader_election.cc:288] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8db739cd4d094f968c637e4a89b325ef (127.3.159.1:44725), 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939)
I20260927 22:01:09.112586 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.102731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:41057 (local address 127.3.159.1:44725)
0927 22:01:09.102988 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:09.102997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:09.104141 (+ 1144us) server_negotiation.cc:407] Connection header received
0927 22:01:09.104408 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:09.104413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:09.104502 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:09.104601 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:09.105543 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.106612 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.109840 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.111184 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.111363 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.111491 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:09.112116 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 22:01:09.112276 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:01:09.112609 5974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.102192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:45525 (local address 127.3.159.3:39939)
0927 22:01:09.102483 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:09.102493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:09.103838 (+ 1345us) server_negotiation.cc:407] Connection header received
0927 22:01:09.104141 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:09.104147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:09.104248 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:09.104375 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:09.105632 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.106768 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.109837 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.111200 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.111406 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.111504 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:09.112109 (+ 605us) server_negotiation.cc:299] Negotiation successful
0927 22:01:09.112276 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 22:01:09.112843 6000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.101707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:39939 (local address 127.3.159.2:45525)
0927 22:01:09.103678 (+ 1971us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:09.103712 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:09.103971 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:09.104737 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:09.104740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:09.104761 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:09.105334 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.105344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.106971 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.106978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.109589 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.109609 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.111603 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.111612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.111850 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.111933 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 22:01:09.112050 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 22:01:09.112274 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1772,"mutex_wait_us":217,"thread_start_us":1530,"threads_started":1}
I20260927 22:01:09.112875 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.101842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:44725 (local address 127.3.159.2:41057)
0927 22:01:09.104011 (+ 2169us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:09.104037 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:09.104230 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:09.104673 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:09.104677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:09.104704 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:09.105268 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.105282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.106778 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.106784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.109593 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.109609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.111612 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.111616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.111852 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.111919 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 22:01:09.112044 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 22:01:09.112274 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1988,"mutex_wait_us":288,"thread_start_us":1744,"threads_started":1}
I20260927 22:01:09.113744 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4353387fbe344609a6c235d221d6de86" candidate_uuid: "ad75e22fd120486ba70562d12a5d6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" is_pre_election: true
I20260927 22:01:09.114075 5914 raft_consensus.cc:2454] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad75e22fd120486ba70562d12a5d6780 in term 0.
I20260927 22:01:09.114308 5634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4353387fbe344609a6c235d221d6de86" candidate_uuid: "ad75e22fd120486ba70562d12a5d6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db739cd4d094f968c637e4a89b325ef" is_pre_election: true
I20260927 22:01:09.114648 5701 leader_election.cc:302] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef955e7a5124be1b5161fd19d48f6f2, ad75e22fd120486ba70562d12a5d6780; no voters:
I20260927 22:01:09.114849 5634 raft_consensus.cc:2454] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad75e22fd120486ba70562d12a5d6780 in term 0.
I20260927 22:01:09.115173 5986 raft_consensus.cc:2781] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:09.115352 5986 raft_consensus.cc:482] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:09.115480 5986 raft_consensus.cc:3037] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:09.118793 5986 raft_consensus.cc:504] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.119537 5986 leader_election.cc:288] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [CANDIDATE]: Term 1 election: Requested vote from peers 8db739cd4d094f968c637e4a89b325ef (127.3.159.1:44725), 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939)
I20260927 22:01:09.120342 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4353387fbe344609a6c235d221d6de86" candidate_uuid: "ad75e22fd120486ba70562d12a5d6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef955e7a5124be1b5161fd19d48f6f2"
I20260927 22:01:09.120374 5634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4353387fbe344609a6c235d221d6de86" candidate_uuid: "ad75e22fd120486ba70562d12a5d6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db739cd4d094f968c637e4a89b325ef"
I20260927 22:01:09.120607 5634 raft_consensus.cc:3037] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:09.120627 5914 raft_consensus.cc:3037] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:09.124271 5634 raft_consensus.cc:2454] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad75e22fd120486ba70562d12a5d6780 in term 1.
I20260927 22:01:09.124255 5914 raft_consensus.cc:2454] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad75e22fd120486ba70562d12a5d6780 in term 1.
I20260927 22:01:09.125257 5700 leader_election.cc:302] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8db739cd4d094f968c637e4a89b325ef, ad75e22fd120486ba70562d12a5d6780; no voters:
I20260927 22:01:09.125883 5986 raft_consensus.cc:2781] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:09.126590 5986 raft_consensus.cc:686] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [term 1 LEADER]: Becoming Leader. State: Replica: ad75e22fd120486ba70562d12a5d6780, State: Running, Role: LEADER
I20260927 22:01:09.127261 5986 consensus_queue.cc:237] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.131354 6002 tablet_replica.cc:439] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: TxnStatusTablet state changed. Reason: New leader ad75e22fd120486ba70562d12a5d6780. Latest consensus state: current_term: 1 leader_uuid: "ad75e22fd120486ba70562d12a5d6780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } }
I20260927 22:01:09.131897 6002 tablet_replica.cc:442] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:01:09.133294 6004 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:01:09.133670 6004 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:01:09.134159 5471 catalog_manager.cc:5495] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad75e22fd120486ba70562d12a5d6780 (127.3.159.2). New cstate: current_term: 1 leader_uuid: "ad75e22fd120486ba70562d12a5d6780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } health_report { overall_health: UNKNOWN } } }
I20260927 22:01:09.169606 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.160556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.3.159.2:34645)
0927 22:01:09.160806 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:09.160816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:09.161229 (+ 413us) server_negotiation.cc:407] Connection header received
0927 22:01:09.161633 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:09.161640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:09.161821 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:09.162054 (+ 233us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:09.162617 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.164008 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.165745 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.166586 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.166803 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.167511 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:09.168561 (+ 1050us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:09.168800 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:09.169023 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 22:01:09.169142 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:01:09.169723 5996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.160555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:34645 (local address 127.0.0.1:35392)
0927 22:01:09.161075 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:09.161115 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:09.161296 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:09.162173 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:09.162178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:09.162192 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:01:09.162443 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.162453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.164294 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.164301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.165517 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.165535 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.166923 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.166930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.167152 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.167252 (+ 100us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:01:09.168789 (+ 1537us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:01:09.168828 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:01:09.168975 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 22:01:09.169259 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 22:01:09.180943 5634 raft_consensus.cc:1261] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Refusing update from remote peer ad75e22fd120486ba70562d12a5d6780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:09.180967 5914 raft_consensus.cc:1261] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Refusing update from remote peer ad75e22fd120486ba70562d12a5d6780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:09.182049 6002 consensus_queue.cc:1035] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:09.182426 5986 consensus_queue.cc:1035] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:09.194243 5985 tablet_replica.cc:439] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef: TxnStatusTablet state changed. Reason: New leader ad75e22fd120486ba70562d12a5d6780. Latest consensus state: current_term: 1 leader_uuid: "ad75e22fd120486ba70562d12a5d6780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } }
I20260927 22:01:09.194641 5985 tablet_replica.cc:442] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:09.198789 6007 mvcc.cc:205] Tried to move back new op lower bound from 7334078337752731648 to 7334078337549447168. Current Snapshot: MvccSnapshot[applied={T|T < 7334078337752731648}]
I20260927 22:01:09.209349 6002 tablet_replica.cc:439] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad75e22fd120486ba70562d12a5d6780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } }
I20260927 22:01:09.209753 6002 tablet_replica.cc:442] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:01:09.214913 5985 tablet_replica.cc:439] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad75e22fd120486ba70562d12a5d6780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } }
I20260927 22:01:09.215324 5985 tablet_replica.cc:442] T 4353387fbe344609a6c235d221d6de86 P 8db739cd4d094f968c637e4a89b325ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:09.218448 5984 tablet_replica.cc:439] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2: TxnStatusTablet state changed. Reason: New leader ad75e22fd120486ba70562d12a5d6780. Latest consensus state: current_term: 1 leader_uuid: "ad75e22fd120486ba70562d12a5d6780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } }
I20260927 22:01:09.218760 5984 tablet_replica.cc:442] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:09.221333 5986 tablet_replica.cc:439] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad75e22fd120486ba70562d12a5d6780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } }
I20260927 22:01:09.221715 5986 tablet_replica.cc:442] T 4353387fbe344609a6c235d221d6de86 P ad75e22fd120486ba70562d12a5d6780: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:01:09.248862 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.238452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.3.159.1:44725)
0927 22:01:09.238692 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:09.238700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:09.238726 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:01:09.238839 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:09.238844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:09.238961 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:09.239146 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:09.239902 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.241342 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.244270 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.245161 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.245322 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.246222 (+ 900us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:09.247309 (+ 1087us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:09.247492 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:09.248287 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 22:01:09.248415 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:01:09.260602 5984 tablet_replica.cc:439] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad75e22fd120486ba70562d12a5d6780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } } }
I20260927 22:01:09.260870 5984 tablet_replica.cc:442] T 4353387fbe344609a6c235d221d6de86 P 6ef955e7a5124be1b5161fd19d48f6f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:01:09.278282 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.270505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:54869 (local address 127.3.159.2:34645)
0927 22:01:09.270763 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:09.270771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:09.271296 (+ 525us) server_negotiation.cc:407] Connection header received
0927 22:01:09.271536 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:09.271544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:09.271669 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:09.271821 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:09.272358 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.273646 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.275459 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.276286 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.276452 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.277180 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:09.277412 (+ 232us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:09.277530 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:09.277764 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 22:01:09.277873 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:01:09.278925 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.270357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:34645 (local address 127.3.159.1:54869)
0927 22:01:09.271177 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:09.271195 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:09.271330 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:09.271854 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:09.271857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:09.271866 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:01:09.272200 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.272209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.273864 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.273869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.275171 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.275186 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.276601 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.276608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.276779 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.276871 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:01:09.277577 (+ 706us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:01:09.277596 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 22:01:09.277710 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 22:01:09.278063 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":457,"threads_started":1}
I20260927 22:01:09.291953 5914 raft_consensus.cc:1261] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Refusing update from remote peer 8db739cd4d094f968c637e4a89b325ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:09.292191 5774 raft_consensus.cc:1261] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Refusing update from remote peer 8db739cd4d094f968c637e4a89b325ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:09.292842 5987 consensus_queue.cc:1035] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:09.293780 5987 consensus_queue.cc:1035] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:09.300961 6008 mvcc.cc:205] Tried to move back new op lower bound from 7334078338211074048 to 7334078336887910400. Current Snapshot: MvccSnapshot[applied={T|T < 7334078338211074048}]
I20260927 22:01:09.349642 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.341327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:45513 (local address 127.3.159.1:44725)
0927 22:01:09.341543 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:09.341548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:09.342532 (+ 984us) server_negotiation.cc:407] Connection header received
0927 22:01:09.342898 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:09.342904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:09.342983 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:09.343084 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:09.344281 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.345238 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.346995 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.347482 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.347619 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.348553 (+ 934us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:09.348777 (+ 224us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:09.348925 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:09.349119 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 22:01:09.349243 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:01:09.349866 6025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.341327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:44725 (local address 127.3.159.2:45513)
0927 22:01:09.342393 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:09.342421 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:09.342745 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:09.343101 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:09.343105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:09.343116 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:01:09.343635 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.343649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.345471 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.345477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.346821 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.346834 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.347751 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.347760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.348041 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.348166 (+ 125us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:01:09.348947 (+ 781us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:01:09.348963 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 22:01:09.349064 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:01:09.349294 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":828,"threads_started":1}
I20260927 22:01:09.414711 5470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:09.415102 5470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:09.420641 5470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5: Sending DeleteTablet for 3 replicas of tablet dc9d2a5963cd401c8cd736b31dd3f0a4
I20260927 22:01:09.422221 5893 tablet_service.cc:1514] Processing DeleteTablet for tablet dc9d2a5963cd401c8cd736b31dd3f0a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:09 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:49592
I20260927 22:01:09.423002 5609 tablet_service.cc:1514] Processing DeleteTablet for tablet dc9d2a5963cd401c8cd736b31dd3f0a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:09 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:36912
I20260927 22:01:09.427197 5470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:01:09.428140 5470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:09.430454 5749 tablet_service.cc:1514] Processing DeleteTablet for tablet dc9d2a5963cd401c8cd736b31dd3f0a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:09 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:35384
I20260927 22:01:09.435868 6028 tablet_replica.cc:330] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: stopping tablet replica
I20260927 22:01:09.436815 6028 raft_consensus.cc:2229] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:09.437881 6028 raft_consensus.cc:2258] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:09.440748 6028 ts_tablet_manager.cc:1905] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:09.446352 5893 tablet_service.cc:1467] Processing CreateTablet for tablet 0191c3d92af3450ea5590e5e77d796b4 (DEFAULT_TABLE table=test-table [id=013ba819470d432ca1aea7156ea681a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:09.447209 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0191c3d92af3450ea5590e5e77d796b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:09.450481 6028 ts_tablet_manager.cc:1918] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:01:09.450837 6028 log.cc:1196] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/dc9d2a5963cd401c8cd736b31dd3f0a4
I20260927 22:01:09.450719 5608 tablet_service.cc:1467] Processing CreateTablet for tablet 0191c3d92af3450ea5590e5e77d796b4 (DEFAULT_TABLE table=test-table [id=013ba819470d432ca1aea7156ea681a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:09.451367 5608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0191c3d92af3450ea5590e5e77d796b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:09.451711 6028 ts_tablet_manager.cc:1939] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting consensus metadata
I20260927 22:01:09.454068 5458 catalog_manager.cc:4841] TS 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939): tablet dc9d2a5963cd401c8cd736b31dd3f0a4 (table test-table [id=4b1f11fca0454db09cdb5962419e9620]) successfully deleted
I20260927 22:01:09.454059 5748 tablet_service.cc:1467] Processing CreateTablet for tablet 0191c3d92af3450ea5590e5e77d796b4 (DEFAULT_TABLE table=test-table [id=013ba819470d432ca1aea7156ea681a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:09.454674 5748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0191c3d92af3450ea5590e5e77d796b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:09.457340 5979 tablet_bootstrap.cc:492] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2: Bootstrap starting.
I20260927 22:01:09.459199 5979 tablet_bootstrap.cc:654] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:09.468981 6030 tablet_replica.cc:330] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780: stopping tablet replica
I20260927 22:01:09.469619 5979 tablet_bootstrap.cc:492] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2: No bootstrap required, opened a new log
I20260927 22:01:09.469702 6030 raft_consensus.cc:2229] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:09.469990 5979 ts_tablet_manager.cc:1397] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260927 22:01:09.470240 6030 raft_consensus.cc:2258] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:09.470655 5980 tablet_bootstrap.cc:492] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef: Bootstrap starting.
I20260927 22:01:09.471063 5979 raft_consensus.cc:348] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.471410 5979 raft_consensus.cc:374] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:09.471531 5979 raft_consensus.cc:729] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef955e7a5124be1b5161fd19d48f6f2, State: Initialized, Role: FOLLOWER
I20260927 22:01:09.471963 6029 tablet_replica.cc:330] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: stopping tablet replica
I20260927 22:01:09.471926 5979 consensus_queue.cc:260] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.472566 6029 raft_consensus.cc:2229] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:01:09.473444 5980 tablet_bootstrap.cc:654] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:09.474812 5979 ts_tablet_manager.cc:1428] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 22:01:09.475286 5978 tablet_bootstrap.cc:492] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780: Bootstrap starting.
I20260927 22:01:09.476501 6030 ts_tablet_manager.cc:1905] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:09.477324 5978 tablet_bootstrap.cc:654] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:09.473469 6029 raft_consensus.cc:2258] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:09.484104 6029 ts_tablet_manager.cc:1905] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:09.491729 5980 tablet_bootstrap.cc:492] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef: No bootstrap required, opened a new log
I20260927 22:01:09.491971 5980 ts_tablet_manager.cc:1397] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.006s
I20260927 22:01:09.492887 5980 raft_consensus.cc:348] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.493171 5980 raft_consensus.cc:374] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:09.493268 5980 raft_consensus.cc:729] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db739cd4d094f968c637e4a89b325ef, State: Initialized, Role: FOLLOWER
I20260927 22:01:09.493604 5980 consensus_queue.cc:260] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.494316 5980 ts_tablet_manager.cc:1428] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 22:01:09.499399 5978 tablet_bootstrap.cc:492] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780: No bootstrap required, opened a new log
I20260927 22:01:09.499665 5978 ts_tablet_manager.cc:1397] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.004s
I20260927 22:01:09.500671 5978 raft_consensus.cc:348] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.501151 5978 raft_consensus.cc:374] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:09.501247 5978 raft_consensus.cc:729] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad75e22fd120486ba70562d12a5d6780, State: Initialized, Role: FOLLOWER
I20260927 22:01:09.501616 5978 consensus_queue.cc:260] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.502444 5978 ts_tablet_manager.cc:1428] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:01:09.506410 6029 ts_tablet_manager.cc:1918] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:01:09.506744 6029 log.cc:1196] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/dc9d2a5963cd401c8cd736b31dd3f0a4
I20260927 22:01:09.507877 6029 ts_tablet_manager.cc:1939] T dc9d2a5963cd401c8cd736b31dd3f0a4 P 8db739cd4d094f968c637e4a89b325ef: Deleting consensus metadata
I20260927 22:01:09.508735 6030 ts_tablet_manager.cc:1918] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:01:09.508936 6030 log.cc:1196] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/dc9d2a5963cd401c8cd736b31dd3f0a4
I20260927 22:01:09.509593 6030 ts_tablet_manager.cc:1939] T dc9d2a5963cd401c8cd736b31dd3f0a4 P ad75e22fd120486ba70562d12a5d6780: Deleting consensus metadata
I20260927 22:01:09.510242 5457 catalog_manager.cc:4841] TS 8db739cd4d094f968c637e4a89b325ef (127.3.159.1:44725): tablet dc9d2a5963cd401c8cd736b31dd3f0a4 (table test-table [id=4b1f11fca0454db09cdb5962419e9620]) successfully deleted
I20260927 22:01:09.513463 5456 catalog_manager.cc:4841] TS ad75e22fd120486ba70562d12a5d6780 (127.3.159.2:34645): tablet dc9d2a5963cd401c8cd736b31dd3f0a4 (table test-table [id=4b1f11fca0454db09cdb5962419e9620]) successfully deleted
I20260927 22:01:09.634502 6002 raft_consensus.cc:482] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:09.634778 6002 raft_consensus.cc:504] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.636504 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191c3d92af3450ea5590e5e77d796b4" candidate_uuid: "ad75e22fd120486ba70562d12a5d6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" is_pre_election: true
I20260927 22:01:09.636579 5634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191c3d92af3450ea5590e5e77d796b4" candidate_uuid: "ad75e22fd120486ba70562d12a5d6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db739cd4d094f968c637e4a89b325ef" is_pre_election: true
I20260927 22:01:09.636795 5914 raft_consensus.cc:2454] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad75e22fd120486ba70562d12a5d6780 in term 0.
I20260927 22:01:09.636971 5634 raft_consensus.cc:2454] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad75e22fd120486ba70562d12a5d6780 in term 0.
I20260927 22:01:09.637100 6002 leader_election.cc:288] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8db739cd4d094f968c637e4a89b325ef (127.3.159.1:44725), 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939)
I20260927 22:01:09.637349 5701 leader_election.cc:302] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef955e7a5124be1b5161fd19d48f6f2, ad75e22fd120486ba70562d12a5d6780; no voters:
I20260927 22:01:09.637737 6002 raft_consensus.cc:2781] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:09.637866 6002 raft_consensus.cc:482] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:09.637987 6002 raft_consensus.cc:3037] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:09.641515 6002 raft_consensus.cc:504] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.643021 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191c3d92af3450ea5590e5e77d796b4" candidate_uuid: "ad75e22fd120486ba70562d12a5d6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef955e7a5124be1b5161fd19d48f6f2"
I20260927 22:01:09.643290 5914 raft_consensus.cc:3037] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:09.643195 5634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191c3d92af3450ea5590e5e77d796b4" candidate_uuid: "ad75e22fd120486ba70562d12a5d6780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db739cd4d094f968c637e4a89b325ef"
I20260927 22:01:09.643498 5634 raft_consensus.cc:3037] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:09.646600 5914 raft_consensus.cc:2454] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad75e22fd120486ba70562d12a5d6780 in term 1.
I20260927 22:01:09.647274 5701 leader_election.cc:302] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef955e7a5124be1b5161fd19d48f6f2, ad75e22fd120486ba70562d12a5d6780; no voters:
I20260927 22:01:09.647672 6014 raft_consensus.cc:2781] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:09.647832 6014 raft_consensus.cc:686] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 1 LEADER]: Becoming Leader. State: Replica: ad75e22fd120486ba70562d12a5d6780, State: Running, Role: LEADER
I20260927 22:01:09.648170 6014 consensus_queue.cc:237] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:09.648993 5634 raft_consensus.cc:2454] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad75e22fd120486ba70562d12a5d6780 in term 1.
I20260927 22:01:09.651177 6002 leader_election.cc:288] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [CANDIDATE]: Term 1 election: Requested vote from peers 8db739cd4d094f968c637e4a89b325ef (127.3.159.1:44725), 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939)
I20260927 22:01:09.653972 5470 catalog_manager.cc:5495] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad75e22fd120486ba70562d12a5d6780 (127.3.159.2). New cstate: current_term: 1 leader_uuid: "ad75e22fd120486ba70562d12a5d6780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } health_report { overall_health: UNKNOWN } } }
I20260927 22:01:09.711892 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.704188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.3.159.2:34645)
0927 22:01:09.704445 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:09.704452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:09.704476 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:01:09.704554 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:09.704559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:09.704665 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:09.704782 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:09.705766 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.706888 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.709348 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.710180 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.710354 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.710940 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:09.711134 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:09.711243 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:09.711429 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 22:01:09.711541 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:01:09.735714 6025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.728970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:34645 (local address 127.3.159.2:48973)
0927 22:01:09.729215 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:09.729240 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:09.729461 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:09.729918 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:09.729925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:09.729941 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:01:09.730264 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.730274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.731443 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.731451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.732609 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:09.732629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.733769 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.733779 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:09.734000 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.734064 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:01:09.734907 (+ 843us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:01:09.734950 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:01:09.735077 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 22:01:09.735284 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":52}
I20260927 22:01:09.736343 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:09.729089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:48973 (local address 127.3.159.2:34645)
0927 22:01:09.729243 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:09.729250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:09.729351 (+ 101us) server_negotiation.cc:407] Connection header received
0927 22:01:09.729608 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:09.729613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:09.729682 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:09.729793 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:09.730437 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.731235 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.732781 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:09.733238 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:09.733659 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:09.734316 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:09.734733 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:09.734907 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:09.735992 (+ 1085us) server_negotiation.cc:299] Negotiation successful
0927 22:01:09.736087 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":61}
I20260927 22:01:09.751434 5914 raft_consensus.cc:1261] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Refusing update from remote peer ad75e22fd120486ba70562d12a5d6780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:09.752115 6002 consensus_queue.cc:1035] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:09.756112 5634 raft_consensus.cc:1261] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Refusing update from remote peer ad75e22fd120486ba70562d12a5d6780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:09.756929 6036 consensus_queue.cc:1035] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:01:09.798761 6007 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:01:11.419394 5470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:11.419683 5470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:11.425825 5470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5: Sending DeleteTablet for 3 replicas of tablet 0191c3d92af3450ea5590e5e77d796b4
I20260927 22:01:11.426946 5748 tablet_service.cc:1514] Processing DeleteTablet for tablet 0191c3d92af3450ea5590e5e77d796b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:11 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:35384
I20260927 22:01:11.426929 5608 tablet_service.cc:1514] Processing DeleteTablet for tablet 0191c3d92af3450ea5590e5e77d796b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:11 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:36912
I20260927 22:01:11.427130 5893 tablet_service.cc:1514] Processing DeleteTablet for tablet 0191c3d92af3450ea5590e5e77d796b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:11 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:49592
I20260927 22:01:11.439731 6063 tablet_replica.cc:330] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2: stopping tablet replica
I20260927 22:01:11.440085 6063 raft_consensus.cc:2229] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:11.440430 6063 raft_consensus.cc:2258] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:11.441699 6062 tablet_replica.cc:330] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef: stopping tablet replica
I20260927 22:01:11.441776 6063 ts_tablet_manager.cc:1905] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:11.442389 6062 raft_consensus.cc:2229] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:11.442916 6062 raft_consensus.cc:2258] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:11.444878 6062 ts_tablet_manager.cc:1905] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:11.448772 6061 tablet_replica.cc:330] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780: stopping tablet replica
I20260927 22:01:11.449102 6061 raft_consensus.cc:2229] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:01:11.449903 6061 raft_consensus.cc:2258] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:11.451207 6061 ts_tablet_manager.cc:1905] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:01:11 dist-test-slave-sg4j krb5kdc[5427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260927 22:01:11.488137 6062 ts_tablet_manager.cc:1918] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:01:11.488332 6062 log.cc:1196] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/0191c3d92af3450ea5590e5e77d796b4
I20260927 22:01:11.488903 6062 ts_tablet_manager.cc:1939] T 0191c3d92af3450ea5590e5e77d796b4 P 8db739cd4d094f968c637e4a89b325ef: Deleting consensus metadata
Sep 27 22:01:11 dist-test-slave-sg4j krb5kdc[5427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:11.513252 6071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:11.493172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.3.159.62:38263)
0927 22:01:11.494165 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:11.494170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:11.494192 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:01:11.494261 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:11.494265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:11.494458 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:11.494635 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:11.496454 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:11.497555 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:11.499093 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:11.499584 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:11.499737 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:11.504688 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:11.504694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:11.504698 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:11.504748 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:11.509569 (+ 4821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:11.510658 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:11.510676 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:11.510684 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:11.510910 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:11.511373 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:11.511378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:11.511381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:11.511751 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:11.512049 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:11.512414 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 22:01:11.512742 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":655,"threads_started":1}
I20260927 22:01:11.519101 5470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:01:11.519969 5470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:11.528949 5608 tablet_service.cc:1467] Processing CreateTablet for tablet 864dfef14126457faf7617fef67359d2 (DEFAULT_TABLE table=test-table [id=5633e35d7fee47f39beeeb6aceaeb4b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:11.529309 5748 tablet_service.cc:1467] Processing CreateTablet for tablet 864dfef14126457faf7617fef67359d2 (DEFAULT_TABLE table=test-table [id=5633e35d7fee47f39beeeb6aceaeb4b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:11.529326 5893 tablet_service.cc:1467] Processing CreateTablet for tablet 864dfef14126457faf7617fef67359d2 (DEFAULT_TABLE table=test-table [id=5633e35d7fee47f39beeeb6aceaeb4b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:11.529520 5608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 864dfef14126457faf7617fef67359d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:11.535192 6072 tablet_bootstrap.cc:492] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef: Bootstrap starting.
I20260927 22:01:11.537496 6072 tablet_bootstrap.cc:654] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:11.540448 6072 tablet_bootstrap.cc:492] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef: No bootstrap required, opened a new log
I20260927 22:01:11.540683 6072 ts_tablet_manager.cc:1397] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 22:01:11.541812 6072 raft_consensus.cc:348] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:11.542057 6072 raft_consensus.cc:374] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:11.542130 6072 raft_consensus.cc:729] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db739cd4d094f968c637e4a89b325ef, State: Initialized, Role: FOLLOWER
I20260927 22:01:11.542433 6072 consensus_queue.cc:260] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:11.543617 6072 ts_tablet_manager.cc:1428] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 22:01:11.556136 5457 catalog_manager.cc:4841] TS 8db739cd4d094f968c637e4a89b325ef (127.3.159.1:44725): tablet 0191c3d92af3450ea5590e5e77d796b4 (table test-table [id=013ba819470d432ca1aea7156ea681a4]) successfully deleted
I20260927 22:01:11.581439 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 864dfef14126457faf7617fef67359d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:11.581568 6063 ts_tablet_manager.cc:1918] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:01:11.581548 6061 ts_tablet_manager.cc:1918] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:01:11.581736 6063 log.cc:1196] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/0191c3d92af3450ea5590e5e77d796b4
I20260927 22:01:11.581755 6061 log.cc:1196] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/0191c3d92af3450ea5590e5e77d796b4
I20260927 22:01:11.582296 6063 ts_tablet_manager.cc:1939] T 0191c3d92af3450ea5590e5e77d796b4 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting consensus metadata
I20260927 22:01:11.582508 6061 ts_tablet_manager.cc:1939] T 0191c3d92af3450ea5590e5e77d796b4 P ad75e22fd120486ba70562d12a5d6780: Deleting consensus metadata
I20260927 22:01:11.583039 5748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 864dfef14126457faf7617fef67359d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:11.584102 5458 catalog_manager.cc:4841] TS 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939): tablet 0191c3d92af3450ea5590e5e77d796b4 (table test-table [id=013ba819470d432ca1aea7156ea681a4]) successfully deleted
I20260927 22:01:11.586242 5456 catalog_manager.cc:4841] TS ad75e22fd120486ba70562d12a5d6780 (127.3.159.2:34645): tablet 0191c3d92af3450ea5590e5e77d796b4 (table test-table [id=013ba819470d432ca1aea7156ea681a4]) successfully deleted
I20260927 22:01:11.590797 6075 tablet_bootstrap.cc:492] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2: Bootstrap starting.
I20260927 22:01:11.592584 6076 tablet_bootstrap.cc:492] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780: Bootstrap starting.
I20260927 22:01:11.592695 6075 tablet_bootstrap.cc:654] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:11.594769 6076 tablet_bootstrap.cc:654] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:11.595053 6075 tablet_bootstrap.cc:492] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2: No bootstrap required, opened a new log
I20260927 22:01:11.595234 6075 ts_tablet_manager.cc:1397] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 22:01:11.596107 6075 raft_consensus.cc:348] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:11.596436 6075 raft_consensus.cc:374] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:11.596539 6075 raft_consensus.cc:729] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef955e7a5124be1b5161fd19d48f6f2, State: Initialized, Role: FOLLOWER
I20260927 22:01:11.596873 6075 consensus_queue.cc:260] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:11.597675 6076 tablet_bootstrap.cc:492] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780: No bootstrap required, opened a new log
I20260927 22:01:11.597867 6075 ts_tablet_manager.cc:1428] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 22:01:11.597879 6076 ts_tablet_manager.cc:1397] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 22:01:11.598824 6076 raft_consensus.cc:348] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:11.599104 6076 raft_consensus.cc:374] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:11.599215 6076 raft_consensus.cc:729] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad75e22fd120486ba70562d12a5d6780, State: Initialized, Role: FOLLOWER
I20260927 22:01:11.599702 6076 consensus_queue.cc:260] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:11.600663 6076 ts_tablet_manager.cc:1428] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 22:01:11.664170 6074 raft_consensus.cc:482] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:11.664578 6074 raft_consensus.cc:504] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:11.665759 6074 leader_election.cc:288] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad75e22fd120486ba70562d12a5d6780 (127.3.159.2:34645), 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939)
I20260927 22:01:11.666591 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864dfef14126457faf7617fef67359d2" candidate_uuid: "8db739cd4d094f968c637e4a89b325ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" is_pre_election: true
I20260927 22:01:11.666637 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864dfef14126457faf7617fef67359d2" candidate_uuid: "8db739cd4d094f968c637e4a89b325ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad75e22fd120486ba70562d12a5d6780" is_pre_election: true
I20260927 22:01:11.666971 5774 raft_consensus.cc:2454] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db739cd4d094f968c637e4a89b325ef in term 0.
I20260927 22:01:11.666972 5914 raft_consensus.cc:2454] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8db739cd4d094f968c637e4a89b325ef in term 0.
I20260927 22:01:11.667659 5561 leader_election.cc:302] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef955e7a5124be1b5161fd19d48f6f2, 8db739cd4d094f968c637e4a89b325ef; no voters:
I20260927 22:01:11.668056 6074 raft_consensus.cc:2781] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:11.668226 6074 raft_consensus.cc:482] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:11.668498 6074 raft_consensus.cc:3037] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:11.690073 6074 raft_consensus.cc:504] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:11.691028 6074 leader_election.cc:288] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [CANDIDATE]: Term 1 election: Requested vote from peers ad75e22fd120486ba70562d12a5d6780 (127.3.159.2:34645), 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939)
I20260927 22:01:11.691795 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864dfef14126457faf7617fef67359d2" candidate_uuid: "8db739cd4d094f968c637e4a89b325ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad75e22fd120486ba70562d12a5d6780"
I20260927 22:01:11.692070 5774 raft_consensus.cc:3037] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:11.692019 5914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "864dfef14126457faf7617fef67359d2" candidate_uuid: "8db739cd4d094f968c637e4a89b325ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ef955e7a5124be1b5161fd19d48f6f2"
I20260927 22:01:11.692273 5914 raft_consensus.cc:3037] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:11.795537 5914 raft_consensus.cc:2454] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db739cd4d094f968c637e4a89b325ef in term 1.
I20260927 22:01:11.795537 5774 raft_consensus.cc:2454] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8db739cd4d094f968c637e4a89b325ef in term 1.
I20260927 22:01:11.796425 5561 leader_election.cc:302] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef955e7a5124be1b5161fd19d48f6f2, 8db739cd4d094f968c637e4a89b325ef; no voters:
I20260927 22:01:11.797029 6074 raft_consensus.cc:2781] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:11.797676 6074 raft_consensus.cc:686] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 1 LEADER]: Becoming Leader. State: Replica: 8db739cd4d094f968c637e4a89b325ef, State: Running, Role: LEADER
I20260927 22:01:11.798162 6074 consensus_queue.cc:237] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:11.802181 5471 catalog_manager.cc:5495] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 8db739cd4d094f968c637e4a89b325ef (127.3.159.1). New cstate: current_term: 1 leader_uuid: "8db739cd4d094f968c637e4a89b325ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } health_report { overall_health: UNKNOWN } } }
I20260927 22:01:11.866278 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:11.858970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36942 (local address 127.3.159.1:44725)
0927 22:01:11.859738 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:11.859743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:11.859767 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:01:11.859849 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:11.859853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:11.860131 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:11.860274 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:11.860859 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:11.861996 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:11.864207 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:11.864729 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:11.864880 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:11.865517 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:11.865651 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:11.865735 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:11.865904 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 22:01:11.865978 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":455,"threads_started":1}
I20260927 22:01:11.877491 5914 raft_consensus.cc:1261] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Refusing update from remote peer 8db739cd4d094f968c637e4a89b325ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:11.877790 5774 raft_consensus.cc:1261] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Refusing update from remote peer 8db739cd4d094f968c637e4a89b325ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:11.878381 6080 consensus_queue.cc:1035] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:11.878831 6074 consensus_queue.cc:1035] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:11.944298 5472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:11.944650 5472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:11.948810 5472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5: Sending DeleteTablet for 3 replicas of tablet 864dfef14126457faf7617fef67359d2
I20260927 22:01:11.950032 5893 tablet_service.cc:1514] Processing DeleteTablet for tablet 864dfef14126457faf7617fef67359d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:11 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:49592
I20260927 22:01:11.950274 5608 tablet_service.cc:1514] Processing DeleteTablet for tablet 864dfef14126457faf7617fef67359d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:11 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:36912
I20260927 22:01:11.950414 6063 tablet_replica.cc:330] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2: stopping tablet replica
I20260927 22:01:11.950667 6063 raft_consensus.cc:2229] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:11.950637 5748 tablet_service.cc:1514] Processing DeleteTablet for tablet 864dfef14126457faf7617fef67359d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:11 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:35384
I20260927 22:01:11.950891 6063 raft_consensus.cc:2258] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:11.950949 6062 tablet_replica.cc:330] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef: stopping tablet replica
I20260927 22:01:11.951143 6061 tablet_replica.cc:330] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780: stopping tablet replica
I20260927 22:01:11.951161 6062 raft_consensus.cc:2229] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:01:11.951416 6061 raft_consensus.cc:2229] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:11.951684 6061 raft_consensus.cc:2258] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:11.951709 6062 raft_consensus.cc:2258] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:11.952893 6063 ts_tablet_manager.cc:1905] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:11.953017 5472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:01:11.953282 6061 ts_tablet_manager.cc:1905] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:11.953294 6062 ts_tablet_manager.cc:1905] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 22:01:11.953879 5472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:11.964237 5608 tablet_service.cc:1467] Processing CreateTablet for tablet b874de8cba4d4fd2982498fa21cc24a3 (DEFAULT_TABLE table=test-table [id=970c2103c0134001806700630a874e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:11.964392 5893 tablet_service.cc:1467] Processing CreateTablet for tablet b874de8cba4d4fd2982498fa21cc24a3 (DEFAULT_TABLE table=test-table [id=970c2103c0134001806700630a874e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:11.964596 5748 tablet_service.cc:1467] Processing CreateTablet for tablet b874de8cba4d4fd2982498fa21cc24a3 (DEFAULT_TABLE table=test-table [id=970c2103c0134001806700630a874e1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:12.002374 5748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b874de8cba4d4fd2982498fa21cc24a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:12.002838 5608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b874de8cba4d4fd2982498fa21cc24a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:12.002986 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b874de8cba4d4fd2982498fa21cc24a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:12.006752 6062 ts_tablet_manager.cc:1918] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:01:12.006979 6062 log.cc:1196] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/864dfef14126457faf7617fef67359d2
I20260927 22:01:12.007470 6062 ts_tablet_manager.cc:1939] T 864dfef14126457faf7617fef67359d2 P 8db739cd4d094f968c637e4a89b325ef: Deleting consensus metadata
I20260927 22:01:12.011346 6075 tablet_bootstrap.cc:492] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2: Bootstrap starting.
I20260927 22:01:12.011641 6072 tablet_bootstrap.cc:492] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef: Bootstrap starting.
I20260927 22:01:12.012010 6061 ts_tablet_manager.cc:1918] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:01:12.012264 6061 log.cc:1196] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/864dfef14126457faf7617fef67359d2
I20260927 22:01:12.012746 6061 ts_tablet_manager.cc:1939] T 864dfef14126457faf7617fef67359d2 P ad75e22fd120486ba70562d12a5d6780: Deleting consensus metadata
I20260927 22:01:12.013527 6076 tablet_bootstrap.cc:492] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780: Bootstrap starting.
I20260927 22:01:12.013715 6063 ts_tablet_manager.cc:1918] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:01:12.013882 6063 log.cc:1196] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/864dfef14126457faf7617fef67359d2
I20260927 22:01:12.014420 6063 ts_tablet_manager.cc:1939] T 864dfef14126457faf7617fef67359d2 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting consensus metadata
I20260927 22:01:12.014456 6075 tablet_bootstrap.cc:654] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:12.014532 6072 tablet_bootstrap.cc:654] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:12.015441 6076 tablet_bootstrap.cc:654] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:12.016990 5458 catalog_manager.cc:4841] TS 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939): tablet 864dfef14126457faf7617fef67359d2 (table test-table [id=5633e35d7fee47f39beeeb6aceaeb4b0]) successfully deleted
I20260927 22:01:12.018326 5456 catalog_manager.cc:4841] TS ad75e22fd120486ba70562d12a5d6780 (127.3.159.2:34645): tablet 864dfef14126457faf7617fef67359d2 (table test-table [id=5633e35d7fee47f39beeeb6aceaeb4b0]) successfully deleted
I20260927 22:01:12.020478 6075 tablet_bootstrap.cc:492] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2: No bootstrap required, opened a new log
I20260927 22:01:12.020691 6075 ts_tablet_manager.cc:1397] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 22:01:12.020956 6076 tablet_bootstrap.cc:492] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780: No bootstrap required, opened a new log
I20260927 22:01:12.021150 6076 ts_tablet_manager.cc:1397] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260927 22:01:12.021613 5457 catalog_manager.cc:4841] TS 8db739cd4d094f968c637e4a89b325ef (127.3.159.1:44725): tablet 864dfef14126457faf7617fef67359d2 (table test-table [id=5633e35d7fee47f39beeeb6aceaeb4b0]) successfully deleted
I20260927 22:01:12.021700 6075 raft_consensus.cc:348] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:12.021931 6072 tablet_bootstrap.cc:492] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef: No bootstrap required, opened a new log
I20260927 22:01:12.021999 6075 raft_consensus.cc:374] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:12.022073 6075 raft_consensus.cc:729] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ef955e7a5124be1b5161fd19d48f6f2, State: Initialized, Role: FOLLOWER
I20260927 22:01:12.022109 6072 ts_tablet_manager.cc:1397] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260927 22:01:12.021955 6076 raft_consensus.cc:348] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:12.022212 6076 raft_consensus.cc:374] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:12.022315 6076 raft_consensus.cc:729] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad75e22fd120486ba70562d12a5d6780, State: Initialized, Role: FOLLOWER
I20260927 22:01:12.022405 6075 consensus_queue.cc:260] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:12.022562 6076 consensus_queue.cc:260] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:12.022917 6072 raft_consensus.cc:348] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:12.023102 6076 ts_tablet_manager.cc:1428] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:01:12.023136 6075 ts_tablet_manager.cc:1428] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:01:12.023211 6072 raft_consensus.cc:374] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:12.023293 6072 raft_consensus.cc:729] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8db739cd4d094f968c637e4a89b325ef, State: Initialized, Role: FOLLOWER
I20260927 22:01:12.023571 6072 consensus_queue.cc:260] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:12.025094 6072 ts_tablet_manager.cc:1428] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 22:01:12.160651 6079 raft_consensus.cc:482] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:12.161018 6079 raft_consensus.cc:504] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:12.163815 6079 leader_election.cc:288] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8db739cd4d094f968c637e4a89b325ef (127.3.159.1:44725), ad75e22fd120486ba70562d12a5d6780 (127.3.159.2:34645)
I20260927 22:01:12.173239 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:12.164347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:49283 (local address 127.3.159.2:34645)
0927 22:01:12.165234 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:12.165239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:12.166128 (+ 889us) server_negotiation.cc:407] Connection header received
0927 22:01:12.166233 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:12.166237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:12.166310 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:12.166445 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:12.167382 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.168264 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.170945 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.171966 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.172154 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:12.172327 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:12.172664 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 22:01:12.172844 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":624,"threads_started":1}
I20260927 22:01:12.173465 6093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:12.164272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:34645 (local address 127.3.159.3:49283)
0927 22:01:12.165091 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:12.165639 (+ 548us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:12.165959 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:12.166733 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:12.166739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:12.166773 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:12.167181 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:12.167195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.168433 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.168439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:12.170715 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:12.170735 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.172243 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.172251 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:12.172456 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:12.172486 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:01:12.172630 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 22:01:12.172824 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":142,"thread_start_us":420,"threads_started":1}
I20260927 22:01:12.173698 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:12.164272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:44725 (local address 127.3.159.3:58515)
0927 22:01:12.166276 (+ 2004us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:12.166299 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:12.166469 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:12.166932 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:12.166940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:12.166971 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:12.167448 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:12.167458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.168577 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.168583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:12.170726 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:12.170742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.172621 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.172629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:12.172813 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:12.172843 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:01:12.172990 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 22:01:12.173151 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1854,"mutex_wait_us":73,"thread_start_us":1563,"threads_started":1}
I20260927 22:01:12.174036 6083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:12.164484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:58515 (local address 127.3.159.1:44725)
0927 22:01:12.164768 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:12.164777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:12.166439 (+ 1662us) server_negotiation.cc:407] Connection header received
0927 22:01:12.166618 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:12.166624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:12.166715 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:12.166825 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:12.167664 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.168417 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.170922 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.172105 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.172563 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:12.172692 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:12.172993 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 22:01:12.173151 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:01:12.174348 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b874de8cba4d4fd2982498fa21cc24a3" candidate_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad75e22fd120486ba70562d12a5d6780" is_pre_election: true
I20260927 22:01:12.174661 5774 raft_consensus.cc:2454] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef955e7a5124be1b5161fd19d48f6f2 in term 0.
I20260927 22:01:12.174695 5633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b874de8cba4d4fd2982498fa21cc24a3" candidate_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db739cd4d094f968c637e4a89b325ef" is_pre_election: true
I20260927 22:01:12.175055 5633 raft_consensus.cc:2454] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ef955e7a5124be1b5161fd19d48f6f2 in term 0.
I20260927 22:01:12.175232 5839 leader_election.cc:302] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef955e7a5124be1b5161fd19d48f6f2, ad75e22fd120486ba70562d12a5d6780; no voters:
I20260927 22:01:12.175752 6079 raft_consensus.cc:2781] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:12.175897 6079 raft_consensus.cc:482] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:12.175968 6079 raft_consensus.cc:3037] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:12.178558 6079 raft_consensus.cc:504] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:12.179226 6079 leader_election.cc:288] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [CANDIDATE]: Term 1 election: Requested vote from peers 8db739cd4d094f968c637e4a89b325ef (127.3.159.1:44725), ad75e22fd120486ba70562d12a5d6780 (127.3.159.2:34645)
I20260927 22:01:12.180060 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b874de8cba4d4fd2982498fa21cc24a3" candidate_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad75e22fd120486ba70562d12a5d6780"
I20260927 22:01:12.180064 5633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b874de8cba4d4fd2982498fa21cc24a3" candidate_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8db739cd4d094f968c637e4a89b325ef"
I20260927 22:01:12.180377 5633 raft_consensus.cc:3037] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:12.180366 5774 raft_consensus.cc:3037] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:12.184930 5774 raft_consensus.cc:2454] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef955e7a5124be1b5161fd19d48f6f2 in term 1.
I20260927 22:01:12.184930 5633 raft_consensus.cc:2454] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ef955e7a5124be1b5161fd19d48f6f2 in term 1.
I20260927 22:01:12.185760 5840 leader_election.cc:302] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ef955e7a5124be1b5161fd19d48f6f2, 8db739cd4d094f968c637e4a89b325ef; no voters:
I20260927 22:01:12.186270 6079 raft_consensus.cc:2781] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:12.187060 6079 raft_consensus.cc:686] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 LEADER]: Becoming Leader. State: Replica: 6ef955e7a5124be1b5161fd19d48f6f2, State: Running, Role: LEADER
I20260927 22:01:12.187667 6079 consensus_queue.cc:237] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } }
I20260927 22:01:12.192306 5470 catalog_manager.cc:5495] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3). New cstate: current_term: 1 leader_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ef955e7a5124be1b5161fd19d48f6f2" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 39939 } health_report { overall_health: HEALTHY } } }
I20260927 22:01:12.236570 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:12.229517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.3.159.3:39939)
0927 22:01:12.230246 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:12.230250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:12.230273 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:01:12.230373 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:12.230377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:12.230463 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:12.230631 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:12.231135 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.231927 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.233711 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.234253 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.234394 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:12.235036 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:12.235723 (+ 687us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:12.235861 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:12.236055 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 22:01:12.236146 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":435,"threads_started":1}
I20260927 22:01:12.259361 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:12.252392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:45221 (local address 127.3.159.2:34645)
0927 22:01:12.252610 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:12.252615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:12.253284 (+ 669us) server_negotiation.cc:407] Connection header received
0927 22:01:12.253571 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:12.253575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:12.253657 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:12.253776 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:12.254398 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.255128 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.256692 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.257122 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.257278 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:12.257845 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:12.258118 (+ 273us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:12.258249 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:12.258865 (+ 616us) server_negotiation.cc:299] Negotiation successful
0927 22:01:12.258970 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:01:12.259474 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:12.252219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:34645 (local address 127.3.159.3:45221)
0927 22:01:12.253151 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:12.253170 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:12.253357 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:12.253855 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:12.253860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:12.253871 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:01:12.254169 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:12.254178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.255284 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.255289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:12.256490 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:12.256503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.257316 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.257324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:12.257576 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:12.257651 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:01:12.258264 (+ 613us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:01:12.258278 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:01:12.258818 (+ 540us) client_negotiation.cc:240] Negotiation successful
0927 22:01:12.259003 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":496,"threads_started":1}
I20260927 22:01:12.270398 5774 raft_consensus.cc:1261] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Refusing update from remote peer 6ef955e7a5124be1b5161fd19d48f6f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:12.270431 5633 raft_consensus.cc:1261] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Refusing update from remote peer 6ef955e7a5124be1b5161fd19d48f6f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:12.271257 6094 consensus_queue.cc:1035] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8db739cd4d094f968c637e4a89b325ef" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:12.271723 6079 consensus_queue.cc:1035] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad75e22fd120486ba70562d12a5d6780" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:12.281065 6044 mvcc.cc:205] Tried to move back new op lower bound from 7334078350410989568 to 7334078350086397952. Current Snapshot: MvccSnapshot[applied={T|T < 7334078350410989568}]
I20260927 22:01:12.307615 6103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:12.300109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:39939 (local address 127.3.159.2:44167)
0927 22:01:12.300866 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:12.300886 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:12.301027 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:12.301487 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:12.301493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:12.301511 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:01:12.301776 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:12.301786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.303248 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.303254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:12.304893 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:12.304911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.306164 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.306172 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:12.306358 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:12.306406 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:01:12.306973 (+ 567us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:01:12.306987 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:01:12.307076 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 22:01:12.307231 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":366,"threads_started":1}
I20260927 22:01:12.308068 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:12.300328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:44167 (local address 127.3.159.3:39939)
0927 22:01:12.300488 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:12.300492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:12.300993 (+ 501us) server_negotiation.cc:407] Connection header received
0927 22:01:12.301160 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:12.301163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:12.301236 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:12.301327 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:12.301983 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.303060 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.305098 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:12.305937 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:12.306074 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:12.306631 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:12.306834 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:12.306927 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:12.307168 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 22:01:12.307253 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 22:01:12.309527 6044 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 22:01:13.960213 5472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:13.960559 5472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:13.966389 5472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66a615240d8247708d6d8c539622a5a5: Sending DeleteTablet for 3 replicas of tablet b874de8cba4d4fd2982498fa21cc24a3
I20260927 22:01:13.967741 5608 tablet_service.cc:1514] Processing DeleteTablet for tablet b874de8cba4d4fd2982498fa21cc24a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:13 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:36912
I20260927 22:01:13.967760 5749 tablet_service.cc:1514] Processing DeleteTablet for tablet b874de8cba4d4fd2982498fa21cc24a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:13 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:35384
I20260927 22:01:13.967923 5893 tablet_service.cc:1514] Processing DeleteTablet for tablet b874de8cba4d4fd2982498fa21cc24a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:13 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:49592
I20260927 22:01:13.968961 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 5543
I20260927 22:01:13.980263 6131 tablet_replica.cc:330] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780: stopping tablet replica
I20260927 22:01:13.980563 6131 raft_consensus.cc:2229] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:13.980612 6132 tablet_replica.cc:330] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef: stopping tablet replica
I20260927 22:01:13.980888 6131 raft_consensus.cc:2258] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:13.981060 6132 raft_consensus.cc:2229] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:13.981798 6132 raft_consensus.cc:2258] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:13.981767 6133 tablet_replica.cc:330] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2: stopping tablet replica
I20260927 22:01:13.982132 6133 raft_consensus.cc:2229] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:01:13.982183 6131 ts_tablet_manager.cc:1905] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:13.982996 6133 raft_consensus.cc:2258] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:13.983265 6132 ts_tablet_manager.cc:1905] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:13.985177 6133 ts_tablet_manager.cc:1905] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:13.990371 6131 ts_tablet_manager.cc:1918] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:01:13.990561 6131 log.cc:1196] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/b874de8cba4d4fd2982498fa21cc24a3
I20260927 22:01:13.991081 6131 ts_tablet_manager.cc:1939] T b874de8cba4d4fd2982498fa21cc24a3 P ad75e22fd120486ba70562d12a5d6780: Deleting consensus metadata
I20260927 22:01:13.992730 5456 catalog_manager.cc:4841] TS ad75e22fd120486ba70562d12a5d6780 (127.3.159.2:34645): tablet b874de8cba4d4fd2982498fa21cc24a3 (table test-table [id=970c2103c0134001806700630a874e1c]) successfully deleted
I20260927 22:01:13.992998 6132 ts_tablet_manager.cc:1918] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:01:13.993152 6132 log.cc:1196] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/b874de8cba4d4fd2982498fa21cc24a3
I20260927 22:01:13.993695 6132 ts_tablet_manager.cc:1939] T b874de8cba4d4fd2982498fa21cc24a3 P 8db739cd4d094f968c637e4a89b325ef: Deleting consensus metadata
I20260927 22:01:13.994874 5457 catalog_manager.cc:4841] TS 8db739cd4d094f968c637e4a89b325ef (127.3.159.1:44725): tablet b874de8cba4d4fd2982498fa21cc24a3 (table test-table [id=970c2103c0134001806700630a874e1c]) successfully deleted
I20260927 22:01:13.995702 6133 ts_tablet_manager.cc:1918] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 22:01:13.995924 6133 log.cc:1196] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/b874de8cba4d4fd2982498fa21cc24a3
I20260927 22:01:13.996464 6133 ts_tablet_manager.cc:1939] T b874de8cba4d4fd2982498fa21cc24a3 P 6ef955e7a5124be1b5161fd19d48f6f2: Deleting consensus metadata
I20260927 22:01:13.998164 5458 catalog_manager.cc:4841] TS 6ef955e7a5124be1b5161fd19d48f6f2 (127.3.159.3:39939): tablet b874de8cba4d4fd2982498fa21cc24a3 (table test-table [id=970c2103c0134001806700630a874e1c]) successfully deleted
Sep 27 22:01:14 dist-test-slave-sg4j krb5kdc[5427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:01:14.012614 6135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:13.971488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.3.159.1:44725)
0927 22:01:13.983998 (+ 12510us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:13.984011 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:13.988179 (+ 4168us) server_negotiation.cc:407] Connection header received
0927 22:01:13.988438 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:13.988444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:13.988555 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:13.988706 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:13.991275 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:13.992790 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:13.995334 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:13.995918 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:13.996131 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:14.004781 (+ 8650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:14.004801 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:14.004820 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:14.004881 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:14.008575 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:14.009649 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:14.009660 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:14.009663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:14.009931 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:14.010590 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:14.010595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:14.010598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:14.011089 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:14.011473 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:14.011869 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 22:01:14.012106 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12264,"thread_start_us":12138,"threads_started":1}
W20260927 22:01:14.013644 5674 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36950
W20260927 22:01:14.014667 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 5543: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:01:14.014971 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 5543
I20260927 22:01:14.042706 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 5683
Sep 27 22:01:14 dist-test-slave-sg4j krb5kdc[5427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:01:14.063835 6136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:14.044202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.3.159.2:34645)
0927 22:01:14.044842 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:14.044847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:14.044870 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:01:14.044934 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:14.044938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:14.045009 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:14.045138 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:14.046687 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:14.047472 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:14.048809 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:14.049604 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:14.049737 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:14.055487 (+ 5750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:14.055508 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:14.055527 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:14.055577 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:14.059974 (+ 4397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:14.061083 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:14.061096 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:14.061099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:14.061311 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:14.062142 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:14.062148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:14.062150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:14.062583 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:14.062853 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:14.063229 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 22:01:14.063436 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":355,"threads_started":1}
W20260927 22:01:14.064656 5814 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35410
W20260927 22:01:14.065551 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 5683: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:01:14.065783 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 5683
I20260927 22:01:14.096421 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 5823
Sep 27 22:01:14 dist-test-slave-sg4j krb5kdc[5427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546471, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:01:14.116006 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:14.097610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49624 (local address 127.3.159.3:39939)
0927 22:01:14.098403 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:14.098410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:14.098442 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:01:14.098509 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:14.098536 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:14.098613 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:14.098733 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:14.100445 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:14.101348 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:14.102987 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:14.103650 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:14.103798 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:14.108983 (+ 5185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:14.109005 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:14.109038 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:14.109087 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:14.112402 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:14.113457 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:14.113471 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:14.113477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:14.113667 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:14.114193 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:14.114198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:14.114200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:14.114607 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:14.114842 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:14.115325 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 22:01:14.115527 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":401,"threads_started":1}
W20260927 22:01:14.116962 5954 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49624
W20260927 22:01:14.117885 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 5823: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:01:14.118088 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 5823
I20260927 22:01:14.143412 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 5440
I20260927 22:01:14.144680 5512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:14.282732 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 5440
2026-09-27T22:01:14Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (11562 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:01:14 dist-test-slave-sg4j krb5kdc[6141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:01:14 dist-test-slave-sg4j krb5kdc[6141](info): set up 2 sockets
Sep 27 22:01:14 dist-test-slave-sg4j krb5kdc[6141](info): commencing operation
krb5kdc: starting...
W20260927 22:01:15.462468 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.008s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:01:15 dist-test-slave-sg4j krb5kdc[6141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:01:15.697790 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:36091
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:36091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:16.124282 6154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:16.124745 6154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:16.124919 6154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:16.133895 6154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:16.134075 6154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:16.134171 6154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:16.134285 6154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:16.134380 6154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:01:16.145526 6154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:36091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:36091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:16.147368 6154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:16.149014 6154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:16.165364 6159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:16.165444 6154 server_base.cc:1034] running on GCE node
W20260927 22:01:16.165603 6160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:16.165541 6162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:16.167519 6154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:16.170351 6154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:16.171741 6154 hybrid_clock.cc:648] HybridClock initialized: now 1790546476171665 us; error 56 us; skew 500 ppm
I20260927 22:01:16.172392 6154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:16 dist-test-slave-sg4j krb5kdc[6141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546476, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:16.177552 6154 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:01:16.179844 6154 webserver.cc:460] Webserver started at http://127.3.159.62:32991/ using document root <none> and password file <none>
I20260927 22:01:16.180874 6154 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:16.181010 6154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:16.181542 6154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:16.185580 6154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "5aef0f1231e742378d4043a5bde8576d"
format_stamp: "Formatted at 2026-09-27 22:01:16 on dist-test-slave-sg4j"
server_key: "3c1db027ad9e72241a5b6dc7a519ca79"
server_key_iv: "1394372b3808241c7d60a77f743627f4"
server_key_version: "encryptionkey@0"
I20260927 22:01:16.186685 6154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "5aef0f1231e742378d4043a5bde8576d"
format_stamp: "Formatted at 2026-09-27 22:01:16 on dist-test-slave-sg4j"
server_key: "3c1db027ad9e72241a5b6dc7a519ca79"
server_key_iv: "1394372b3808241c7d60a77f743627f4"
server_key_version: "encryptionkey@0"
I20260927 22:01:16.194420 6154 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:01:16.199532 6169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:16.201702 6154 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:01:16.202018 6154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "5aef0f1231e742378d4043a5bde8576d"
format_stamp: "Formatted at 2026-09-27 22:01:16 on dist-test-slave-sg4j"
server_key: "3c1db027ad9e72241a5b6dc7a519ca79"
server_key_iv: "1394372b3808241c7d60a77f743627f4"
server_key_version: "encryptionkey@0"
I20260927 22:01:16.202375 6154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:16.223366 6154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:16.229674 6154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:16.230067 6154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:16.255046 6154 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:36091
I20260927 22:01:16.255092 6220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:36091 every 8 connection(s)
I20260927 22:01:16.257054 6154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:16.262036 6221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:16.264043 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 6154
I20260927 22:01:16.264451 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:16.265184 3708 external_mini_cluster.cc:1428] Setting key 16379a0d87b4580e307147ed8f33e053
I20260927 22:01:16.276824 6221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d: Bootstrap starting.
I20260927 22:01:16.282354 6221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:16.283994 6221 log.cc:824] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:01:16 dist-test-slave-sg4j krb5kdc[6141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:16.288012 6221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d: No bootstrap required, opened a new log
I20260927 22:01:16.296579 6224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:16.267838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.3.159.62:36091)
0927 22:01:16.268955 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:16.268971 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:16.269030 (+ 59us) server_negotiation.cc:407] Connection header received
0927 22:01:16.271382 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:16.271427 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:16.272164 (+ 737us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:16.272638 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:16.274837 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:16.276461 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:16.278723 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:16.279674 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:16.279864 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:16.286130 (+ 6266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:16.286141 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:16.286167 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:16.286225 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:16.290606 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:16.291773 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:16.291786 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:16.291796 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:16.292049 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:16.292589 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:16.292594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:16.292596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:16.293522 (+ 926us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:16.293886 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:16.294404 (+ 518us) server_negotiation.cc:299] Negotiation successful
0927 22:01:16.295065 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":505,"threads_started":1}
I20260927 22:01:16.298097 6221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aef0f1231e742378d4043a5bde8576d" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36091 } }
I20260927 22:01:16.298583 6221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:16.298712 6221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aef0f1231e742378d4043a5bde8576d, State: Initialized, Role: FOLLOWER
I20260927 22:01:16.299485 6221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aef0f1231e742378d4043a5bde8576d" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36091 } }
I20260927 22:01:16.299830 6221 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:16.299997 6221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:16.300207 6221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:16.303359 6221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aef0f1231e742378d4043a5bde8576d" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36091 } }
I20260927 22:01:16.304050 6221 leader_election.cc:302] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aef0f1231e742378d4043a5bde8576d; no voters:
I20260927 22:01:16.304749 6221 leader_election.cc:288] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:16.305035 6226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:16.306592 6226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [term 1 LEADER]: Becoming Leader. State: Replica: 5aef0f1231e742378d4043a5bde8576d, State: Running, Role: LEADER
I20260927 22:01:16.307395 6226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aef0f1231e742378d4043a5bde8576d" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36091 } }
I20260927 22:01:16.307731 6221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:16.311532 6227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aef0f1231e742378d4043a5bde8576d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aef0f1231e742378d4043a5bde8576d" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36091 } } }
I20260927 22:01:16.311661 6228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aef0f1231e742378d4043a5bde8576d. Latest consensus state: current_term: 1 leader_uuid: "5aef0f1231e742378d4043a5bde8576d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aef0f1231e742378d4043a5bde8576d" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36091 } } }
I20260927 22:01:16.312244 6227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:16.312282 6228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:16.313752 6233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:16.321693 6233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:16.339958 6233 catalog_manager.cc:1347] Generated new cluster ID: a2e010118f83480694011d626563be7a
I20260927 22:01:16.340142 6233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:16.353154 6233 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:16.354866 6233 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:16.363739 6233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aef0f1231e742378d4043a5bde8576d: Generated new TSK 0
I20260927 22:01:16.365625 6233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:01:16.438454 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36091
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:16.876792 6247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:16.877166 6247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:16.877360 6247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:16.885500 6247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:16.885640 6247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:16.885773 6247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:01:16.896912 6247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:16.898661 6247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:16.900344 6247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:16.919497 6252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:16.919535 6253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:16.919607 6255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:16.920660 6247 server_base.cc:1034] running on GCE node
I20260927 22:01:16.921495 6247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:16.922648 6247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:16.923951 6247 hybrid_clock.cc:648] HybridClock initialized: now 1790546476923862 us; error 87 us; skew 500 ppm
I20260927 22:01:16.924499 6247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:16 dist-test-slave-sg4j krb5kdc[6141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546476, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:16.928741 6247 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:01:16.930954 6247 webserver.cc:460] Webserver started at http://127.3.159.1:45129/ using document root <none> and password file <none>
I20260927 22:01:16.931924 6247 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:16.932094 6247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:16.932485 6247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:16.936368 6247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "db0d96dbf93c4c6e9b1e7486747b0016"
format_stamp: "Formatted at 2026-09-27 22:01:16 on dist-test-slave-sg4j"
server_key: "33ebbc23599cb1e805edf87810d82791"
server_key_iv: "af7b74f01c1265b45317161f7c075ade"
server_key_version: "encryptionkey@0"
I20260927 22:01:16.937317 6247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "db0d96dbf93c4c6e9b1e7486747b0016"
format_stamp: "Formatted at 2026-09-27 22:01:16 on dist-test-slave-sg4j"
server_key: "33ebbc23599cb1e805edf87810d82791"
server_key_iv: "af7b74f01c1265b45317161f7c075ade"
server_key_version: "encryptionkey@0"
I20260927 22:01:16.944255 6247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:01:16.949545 6262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:16.951526 6247 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:01:16.951843 6247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "db0d96dbf93c4c6e9b1e7486747b0016"
format_stamp: "Formatted at 2026-09-27 22:01:16 on dist-test-slave-sg4j"
server_key: "33ebbc23599cb1e805edf87810d82791"
server_key_iv: "af7b74f01c1265b45317161f7c075ade"
server_key_version: "encryptionkey@0"
I20260927 22:01:16.952123 6247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:16.970005 6247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:16.975238 6247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:16.975610 6247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:16.976864 6247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:16.978513 6247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:16.978623 6247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:16.978791 6247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:16.978873 6247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:17.022156 6247 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:41775
I20260927 22:01:17.022313 6374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:41775 every 8 connection(s)
I20260927 22:01:17.023947 6247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:01:17.027176 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 6247
I20260927 22:01:17.027433 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:01:17.028273 3708 external_mini_cluster.cc:1428] Setting key 19c1960973b69bc22fc7d2523af20dbb
Sep 27 22:01:17 dist-test-slave-sg4j krb5kdc[6141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546476, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:01:17.053761 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:17.027430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:34651 (local address 127.3.159.62:36091)
0927 22:01:17.028096 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:17.028103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:17.029432 (+ 1329us) server_negotiation.cc:407] Connection header received
0927 22:01:17.031673 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:17.031683 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:17.031802 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:17.031969 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:17.034173 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:17.035494 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:17.037726 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:17.038217 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:17.038399 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:17.044455 (+ 6056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:17.044466 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:17.044475 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:17.044552 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:17.048884 (+ 4332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:17.050324 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:17.050333 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:17.050339 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:17.050554 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:17.051166 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:17.051174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:17.051179 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:17.051653 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:17.051849 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:17.053060 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0927 22:01:17.053290 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":346,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:01:17.055619 6378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:17.028050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:36091 (local address 127.3.159.1:34651)
0927 22:01:17.029174 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:17.029236 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:17.031257 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:17.032319 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:17.032328 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:17.033005 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:17.033874 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:17.033898 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:17.035797 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:17.035803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:17.037455 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:17.037495 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:17.038532 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:17.038540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:17.038997 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:17.040004 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:17.040045 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:17.043945 (+ 3900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:17.049224 (+ 5279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:17.049241 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:17.049268 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:17.050080 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:17.050754 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:17.050758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:17.050761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:17.050998 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:17.051872 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:17.051881 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:17.052460 (+ 579us) client_negotiation.cc:769] Sending connection context
0927 22:01:17.052892 (+ 432us) client_negotiation.cc:240] Negotiation successful
0927 22:01:17.053676 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":438,"threads_started":1}
I20260927 22:01:17.057945 6375 heartbeater.cc:347] Connected to a master server at 127.3.159.62:36091
I20260927 22:01:17.058415 6375 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:17.059391 6375 heartbeater.cc:511] Master 127.3.159.62:36091 requested a full tablet report, sending...
I20260927 22:01:17.062865 6186 ts_manager.cc:194] Registered new tserver with Master: db0d96dbf93c4c6e9b1e7486747b0016 (127.3.159.1:41775)
I20260927 22:01:17.064939 6186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:34651
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:01:17.086938 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36091
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:17.479984 6381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:17.480427 6381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:17.480600 6381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:17.489090 6381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:17.489302 6381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:17.489542 6381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:01:17.500379 6381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:17.502262 6381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:17.503886 6381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:17.520038 6387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:17.520089 6389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:17.520136 6386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:17.520956 6381 server_base.cc:1034] running on GCE node
I20260927 22:01:17.521831 6381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:17.523069 6381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:17.524379 6381 hybrid_clock.cc:648] HybridClock initialized: now 1790546477524317 us; error 51 us; skew 500 ppm
I20260927 22:01:17.524820 6381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:17 dist-test-slave-sg4j krb5kdc[6141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546477, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:17.528972 6381 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:01:17.531296 6381 webserver.cc:460] Webserver started at http://127.3.159.2:45809/ using document root <none> and password file <none>
I20260927 22:01:17.532292 6381 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:17.532465 6381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:17.532892 6381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:17.537012 6381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "6e600fb3df494a9bac798d2eca40e1be"
format_stamp: "Formatted at 2026-09-27 22:01:17 on dist-test-slave-sg4j"
server_key: "27027cf54d44f0a812c4e070c78c721e"
server_key_iv: "4aaba2dc69ec83e1c6cb317ff2eb33fb"
server_key_version: "encryptionkey@0"
I20260927 22:01:17.538206 6381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "6e600fb3df494a9bac798d2eca40e1be"
format_stamp: "Formatted at 2026-09-27 22:01:17 on dist-test-slave-sg4j"
server_key: "27027cf54d44f0a812c4e070c78c721e"
server_key_iv: "4aaba2dc69ec83e1c6cb317ff2eb33fb"
server_key_version: "encryptionkey@0"
I20260927 22:01:17.545924 6381 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 22:01:17.551023 6396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:17.553323 6381 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:01:17.553691 6381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "6e600fb3df494a9bac798d2eca40e1be"
format_stamp: "Formatted at 2026-09-27 22:01:17 on dist-test-slave-sg4j"
server_key: "27027cf54d44f0a812c4e070c78c721e"
server_key_iv: "4aaba2dc69ec83e1c6cb317ff2eb33fb"
server_key_version: "encryptionkey@0"
I20260927 22:01:17.553999 6381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:17.577606 6381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:17.583613 6381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:17.584026 6381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:17.585476 6381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:17.587397 6381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:17.587491 6381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:17.587641 6381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:17.587715 6381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:17.630204 6381 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:33387
I20260927 22:01:17.630275 6508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:33387 every 8 connection(s)
I20260927 22:01:17.631946 6381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:01:17.633960 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 6381
I20260927 22:01:17.634193 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:01:17.634737 3708 external_mini_cluster.cc:1428] Setting key 0d2856df676eda8238eeca5aeda65834
Sep 27 22:01:17 dist-test-slave-sg4j krb5kdc[6141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546477, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
I20260927 22:01:17.667243 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:17.635533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:34829 (local address 127.3.159.62:36091)
0927 22:01:17.641881 (+ 6348us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:17.641887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:17.641915 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:01:17.642405 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:17.642409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:17.642496 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:17.642636 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:17.645818 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:17.647065 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:17.650336 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:17.650844 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:17.650968 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:17.656660 (+ 5692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:17.656671 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:17.656678 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:17.656739 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:17.661446 (+ 4707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:17.663321 (+ 1875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:17.663329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:17.663332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:17.663525 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:17.664671 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:17.664679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:17.664684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:17.665153 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:17.665336 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:17.666488 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0927 22:01:17.666887 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6223,"thread_start_us":6090,"threads_started":1}
I20260927 22:01:17.668535 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:17.636234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:36091 (local address 127.3.159.2:34829)
0927 22:01:17.638321 (+ 2087us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:17.638417 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:17.641945 (+ 3528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:17.643060 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:17.643080 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:17.644272 (+ 1192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:17.645452 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:17.645492 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:17.647335 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:17.647345 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:17.650085 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:17.650108 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:17.651171 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:17.651181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:17.651584 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:17.652528 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:17.652577 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:17.656098 (+ 3521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:17.662356 (+ 6258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:17.662367 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:17.662389 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:17.663019 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:17.663816 (+ 797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:17.663826 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:17.663833 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:17.664275 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:17.665429 (+ 1154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:17.665449 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:17.665900 (+ 451us) client_negotiation.cc:769] Sending connection context
0927 22:01:17.666264 (+ 364us) client_negotiation.cc:240] Negotiation successful
0927 22:01:17.666898 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1595,"thread_start_us":1239,"threads_started":1}
I20260927 22:01:17.670624 6509 heartbeater.cc:347] Connected to a master server at 127.3.159.62:36091
I20260927 22:01:17.671106 6509 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:17.672204 6509 heartbeater.cc:511] Master 127.3.159.62:36091 requested a full tablet report, sending...
I20260927 22:01:17.674307 6186 ts_manager.cc:194] Registered new tserver with Master: 6e600fb3df494a9bac798d2eca40e1be (127.3.159.2:33387)
I20260927 22:01:17.675263 6186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:34829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:01:17.698899 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36091
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:18.070267 6375 heartbeater.cc:503] Master 127.3.159.62:36091 was elected leader, sending a full tablet report...
W20260927 22:01:18.113765 6515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:18.114159 6515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:18.114272 6515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:18.123718 6515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:18.123916 6515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:18.124133 6515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:01:18.136816 6515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36091
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:18.138680 6515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:18.140453 6515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:18.157639 6523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:18.157517 6521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:18.157483 6520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:18.159824 6515 server_base.cc:1034] running on GCE node
I20260927 22:01:18.160848 6515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:18.162622 6515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:18.164069 6515 hybrid_clock.cc:648] HybridClock initialized: now 1790546478164001 us; error 91 us; skew 500 ppm
I20260927 22:01:18.164626 6515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:18 dist-test-slave-sg4j krb5kdc[6141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546478, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:18.169551 6515 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:01:18.171725 6515 webserver.cc:460] Webserver started at http://127.3.159.3:35329/ using document root <none> and password file <none>
I20260927 22:01:18.172699 6515 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:18.172873 6515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:18.173260 6515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:18.177608 6515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "2a034d5bea5147bf82e222bc1ff1ec0c"
format_stamp: "Formatted at 2026-09-27 22:01:18 on dist-test-slave-sg4j"
server_key: "979c66fccc24a0118f60e0ced2667553"
server_key_iv: "393a9716712820524216f0db4840be01"
server_key_version: "encryptionkey@0"
I20260927 22:01:18.178795 6515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "2a034d5bea5147bf82e222bc1ff1ec0c"
format_stamp: "Formatted at 2026-09-27 22:01:18 on dist-test-slave-sg4j"
server_key: "979c66fccc24a0118f60e0ced2667553"
server_key_iv: "393a9716712820524216f0db4840be01"
server_key_version: "encryptionkey@0"
I20260927 22:01:18.186321 6515 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:01:18.191484 6530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:18.193605 6515 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260927 22:01:18.193864 6515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "2a034d5bea5147bf82e222bc1ff1ec0c"
format_stamp: "Formatted at 2026-09-27 22:01:18 on dist-test-slave-sg4j"
server_key: "979c66fccc24a0118f60e0ced2667553"
server_key_iv: "393a9716712820524216f0db4840be01"
server_key_version: "encryptionkey@0"
I20260927 22:01:18.194159 6515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:18.210623 6515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:18.216573 6515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:18.216945 6515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:18.218375 6515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:18.220067 6515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:18.220180 6515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:18.220357 6515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:18.220438 6515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:18.264297 6515 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:40867
I20260927 22:01:18.264384 6642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:40867 every 8 connection(s)
I20260927 22:01:18.266311 6515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:01:18.275046 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 6515
I20260927 22:01:18.275502 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:01:18.276190 3708 external_mini_cluster.cc:1428] Setting key bdb64cd6e60e8a3ba54acae4f84c5f79
Sep 27 22:01:18 dist-test-slave-sg4j krb5kdc[6141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546478, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:18.295070 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:18.269817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:34257 (local address 127.3.159.62:36091)
0927 22:01:18.270366 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:18.270372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:18.271710 (+ 1338us) server_negotiation.cc:407] Connection header received
0927 22:01:18.274076 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:18.274083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:18.274215 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:18.274357 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:18.277273 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:18.278366 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:18.281218 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:18.281770 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:18.281923 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:18.287245 (+ 5322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:18.287251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:18.287255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:18.287304 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:18.290307 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:18.291557 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:18.291564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:18.291568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:18.291775 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:18.292432 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:18.292436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:18.292438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:18.292898 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:18.293064 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:18.294528 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0927 22:01:18.294774 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":299,"threads_started":1}
I20260927 22:01:18.296293 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:18.270461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:36091 (local address 127.3.159.3:34257)
0927 22:01:18.271459 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:18.271518 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:18.273707 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:18.274722 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:18.274737 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:18.275909 (+ 1172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:18.277046 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:18.277073 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:18.278619 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:18.278625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:18.280998 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:18.281019 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:18.282102 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:18.282110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:18.282449 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:18.283532 (+ 1083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:18.283574 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:18.286857 (+ 3283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:18.290560 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:18.290572 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:18.290594 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:18.291317 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:18.291937 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:18.291942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:18.291945 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:18.292290 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:18.293080 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:18.293087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:18.293888 (+ 801us) client_negotiation.cc:769] Sending connection context
0927 22:01:18.294332 (+ 444us) client_negotiation.cc:240] Negotiation successful
0927 22:01:18.294925 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":375,"threads_started":1}
I20260927 22:01:18.298355 6643 heartbeater.cc:347] Connected to a master server at 127.3.159.62:36091
I20260927 22:01:18.298785 6643 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:18.299798 6643 heartbeater.cc:511] Master 127.3.159.62:36091 requested a full tablet report, sending...
I20260927 22:01:18.301734 6186 ts_manager.cc:194] Registered new tserver with Master: 2a034d5bea5147bf82e222bc1ff1ec0c (127.3.159.3:40867)
I20260927 22:01:18.302750 6186 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:34257
I20260927 22:01:18.307575 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:01:18.346513 6645 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:01:18.343551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.3.159.62:36091)
0927 22:01:18.343762 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:18.343767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:18.344354 (+ 587us) server_negotiation.cc:407] Connection header received
0927 22:01:18.344517 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:18.344521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:18.344603 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:18.344728 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:18.346301 (+ 1573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47290: BlockingRecv error: recv got EOF from 127.0.0.1:47290 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:01:18.351999 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 6247
W20260927 22:01:18.355664 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 6247: Not authorized: Client connection negotiation failed: client connection to 127.3.159.1:41775: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:01:18.355973 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 6247
I20260927 22:01:18.373927 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 6381
W20260927 22:01:18.377758 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 6381: Not authorized: Client connection negotiation failed: client connection to 127.3.159.2:33387: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:01:18.377962 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 6381
W20260927 22:01:18.378046 6656 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:01:18.375291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.3.159.2:33387)
0927 22:01:18.375950 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:18.375960 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:18.375993 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:01:18.376071 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:18.376079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:18.376425 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:18.376647 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:18.377579 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38344: BlockingRecv error: recv got EOF from 127.0.0.1:38344 (error 108)
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":323,"threads_started":1}
I20260927 22:01:18.394735 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 6515
W20260927 22:01:18.399463 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 6515: Not authorized: Client connection negotiation failed: client connection to 127.3.159.3:40867: server requires authentication, but client does not have Kerberos credentials available
I20260927 22:01:18.399801 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 6515
I20260927 22:01:18.417800 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 6154
I20260927 22:01:18.419360 6216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:18.537573 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 6154
2026-09-27T22:01:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4249 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:01:18 dist-test-slave-sg4j krb5kdc[6661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:01:18 dist-test-slave-sg4j krb5kdc[6661](info): set up 2 sockets
Sep 27 22:01:18 dist-test-slave-sg4j krb5kdc[6661](info): commencing operation
krb5kdc: starting...
W20260927 22:01:19.670579 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:01:19 dist-test-slave-sg4j krb5kdc[6661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:01:19.923454 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:43527
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:43527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:20.395705 6674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:20.396138 6674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:20.396245 6674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:20.405151 6674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:20.405351 6674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:20.405507 6674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:20.405592 6674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:20.405651 6674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:01:20.418819 6674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:43527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:43527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:20.420791 6674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:20.422726 6674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:20.439759 6680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:20.439749 6679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:20.440279 6682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:20.441982 6674 server_base.cc:1034] running on GCE node
I20260927 22:01:20.443259 6674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:20.445809 6674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:20.447148 6674 hybrid_clock.cc:648] HybridClock initialized: now 1790546480447075 us; error 128 us; skew 500 ppm
I20260927 22:01:20.447798 6674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:20 dist-test-slave-sg4j krb5kdc[6661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:20.457499 6674 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:01:20.460510 6674 webserver.cc:460] Webserver started at http://127.3.159.62:43887/ using document root <none> and password file <none>
I20260927 22:01:20.461820 6674 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:20.462042 6674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:20.462495 6674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:20.466984 6674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "a989a82f7884474289bc5c5009eea662"
format_stamp: "Formatted at 2026-09-27 22:01:20 on dist-test-slave-sg4j"
server_key: "93e0c750464d2e2ffca55691ab3b66a9"
server_key_iv: "4c09fe3b9dbe29d6c705bd47ec58677c"
server_key_version: "encryptionkey@0"
I20260927 22:01:20.468135 6674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "a989a82f7884474289bc5c5009eea662"
format_stamp: "Formatted at 2026-09-27 22:01:20 on dist-test-slave-sg4j"
server_key: "93e0c750464d2e2ffca55691ab3b66a9"
server_key_iv: "4c09fe3b9dbe29d6c705bd47ec58677c"
server_key_version: "encryptionkey@0"
I20260927 22:01:20.476229 6674 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260927 22:01:20.481894 6689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:20.484653 6674 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260927 22:01:20.484918 6674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "a989a82f7884474289bc5c5009eea662"
format_stamp: "Formatted at 2026-09-27 22:01:20 on dist-test-slave-sg4j"
server_key: "93e0c750464d2e2ffca55691ab3b66a9"
server_key_iv: "4c09fe3b9dbe29d6c705bd47ec58677c"
server_key_version: "encryptionkey@0"
I20260927 22:01:20.485203 6674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:20.505897 6674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:20.511456 6674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:20.511862 6674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:20.536988 6674 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:43527
I20260927 22:01:20.537029 6740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:43527 every 8 connection(s)
I20260927 22:01:20.538955 6674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:20.544544 6741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:20.546293 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 6674
I20260927 22:01:20.546571 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:20.547302 3708 external_mini_cluster.cc:1428] Setting key b9caed7a6c670405d68f7cbb81114c83
I20260927 22:01:20.562796 6741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662: Bootstrap starting.
Sep 27 22:01:20 dist-test-slave-sg4j krb5kdc[6661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:20.567407 6741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:20.568688 6741 log.cc:824] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:20.572943 6741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662: No bootstrap required, opened a new log
I20260927 22:01:20.578063 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:20.549902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36370 (local address 127.3.159.62:43527)
0927 22:01:20.551013 (+ 1111us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:20.551030 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:20.551094 (+ 64us) server_negotiation.cc:407] Connection header received
0927 22:01:20.553089 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:20.553123 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:20.553876 (+ 753us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:20.554418 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:20.556821 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:20.558585 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:20.560672 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:20.561778 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:20.562004 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:20.568079 (+ 6075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:20.568093 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:20.568118 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:20.568186 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:20.572509 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:20.573627 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:20.573636 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:20.573646 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:20.573849 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:20.574312 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:20.574316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:20.574323 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:20.575090 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:20.575507 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:20.575962 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 22:01:20.576526 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":501,"threads_started":1}
I20260927 22:01:20.582530 6741 raft_consensus.cc:348] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a989a82f7884474289bc5c5009eea662" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43527 } }
I20260927 22:01:20.583030 6741 raft_consensus.cc:374] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:20.583178 6741 raft_consensus.cc:729] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a989a82f7884474289bc5c5009eea662, State: Initialized, Role: FOLLOWER
I20260927 22:01:20.583707 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:01:20.584292 6741 consensus_queue.cc:260] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a989a82f7884474289bc5c5009eea662" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43527 } }
I20260927 22:01:20.584686 6741 raft_consensus.cc:388] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:20.584861 6741 raft_consensus.cc:482] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:20.585073 6741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:20.588147 6741 raft_consensus.cc:504] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a989a82f7884474289bc5c5009eea662" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43527 } }
I20260927 22:01:20.589202 6741 leader_election.cc:302] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a989a82f7884474289bc5c5009eea662; no voters:
I20260927 22:01:20.590233 6741 leader_election.cc:288] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:20.590417 6746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:20.591997 6746 raft_consensus.cc:686] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [term 1 LEADER]: Becoming Leader. State: Replica: a989a82f7884474289bc5c5009eea662, State: Running, Role: LEADER
I20260927 22:01:20.592871 6746 consensus_queue.cc:237] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a989a82f7884474289bc5c5009eea662" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43527 } }
I20260927 22:01:20.593144 6741 sys_catalog.cc:564] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:20.596565 6748 sys_catalog.cc:455] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a989a82f7884474289bc5c5009eea662. Latest consensus state: current_term: 1 leader_uuid: "a989a82f7884474289bc5c5009eea662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a989a82f7884474289bc5c5009eea662" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43527 } } }
I20260927 22:01:20.597072 6748 sys_catalog.cc:458] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:20.597720 6747 sys_catalog.cc:455] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a989a82f7884474289bc5c5009eea662" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a989a82f7884474289bc5c5009eea662" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43527 } } }
I20260927 22:01:20.598076 6747 sys_catalog.cc:458] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:20.607504 6753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:20.614646 6753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:20.624361 6765 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:01:20.628579 6753 catalog_manager.cc:1347] Generated new cluster ID: 89ffaa13048840918320c55f752573c1
I20260927 22:01:20.628726 6753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:20.653146 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:20.636113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36374 (local address 127.3.159.62:43527)
0927 22:01:20.636315 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:20.636322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:20.642506 (+ 6184us) server_negotiation.cc:407] Connection header received
0927 22:01:20.642898 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:20.642905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:20.643026 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:20.643208 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:20.644620 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:20.646042 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:20.648165 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:20.648976 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:20.649181 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:20.650720 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:20.650728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:20.650734 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:20.651966 (+ 1232us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:20.652076 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:20.652124 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:20.652272 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:20.652501 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 22:01:20.652640 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":70}
I20260927 22:01:20.654551 6753 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:20.656023 6753 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:20.669816 6753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a989a82f7884474289bc5c5009eea662: Generated new TSK 0
I20260927 22:01:20.671278 6753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:01:20.688822 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 6674
I20260927 22:01:20.691115 6736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:20.800765 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 6674
2026-09-27T22:01:20Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2259 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:01:20 dist-test-slave-sg4j krb5kdc[6775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:01:20 dist-test-slave-sg4j krb5kdc[6775](info): set up 2 sockets
Sep 27 22:01:20 dist-test-slave-sg4j krb5kdc[6775](info): commencing operation
krb5kdc: starting...
W20260927 22:01:21.958668 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:01:22 dist-test-slave-sg4j krb5kdc[6775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:01:22.198899 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:46295
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:46295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:22.598742 6788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:22.599156 6788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:22.599309 6788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:22.608053 6788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:22.608245 6788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:22.608348 6788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:22.608487 6788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:22.608592 6788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:01:22.621127 6788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:46295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:46295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:22.622972 6788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:22.624609 6788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:22.639729 6794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:22.639771 6793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:22.640695 6796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:22.640749 6788 server_base.cc:1034] running on GCE node
I20260927 22:01:22.641863 6788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:22.643748 6788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:22.645056 6788 hybrid_clock.cc:648] HybridClock initialized: now 1790546482644953 us; error 86 us; skew 500 ppm
I20260927 22:01:22.645733 6788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:22 dist-test-slave-sg4j krb5kdc[6775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546482, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:22.650424 6788 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:01:22.652560 6788 webserver.cc:460] Webserver started at http://127.3.159.62:33893/ using document root <none> and password file <none>
I20260927 22:01:22.653308 6788 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:22.653548 6788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:22.653879 6788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:22.657104 6788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "17df232b4bae48dda942df172351830e"
format_stamp: "Formatted at 2026-09-27 22:01:22 on dist-test-slave-sg4j"
server_key: "aaa5665dd2e7220eb712433880999c02"
server_key_iv: "e4f272aab6b137120172fd6eaab1e6e0"
server_key_version: "encryptionkey@0"
I20260927 22:01:22.657997 6788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "17df232b4bae48dda942df172351830e"
format_stamp: "Formatted at 2026-09-27 22:01:22 on dist-test-slave-sg4j"
server_key: "aaa5665dd2e7220eb712433880999c02"
server_key_iv: "e4f272aab6b137120172fd6eaab1e6e0"
server_key_version: "encryptionkey@0"
I20260927 22:01:22.664402 6788 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:01:22.669072 6803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:22.671360 6788 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 22:01:22.671725 6788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "17df232b4bae48dda942df172351830e"
format_stamp: "Formatted at 2026-09-27 22:01:22 on dist-test-slave-sg4j"
server_key: "aaa5665dd2e7220eb712433880999c02"
server_key_iv: "e4f272aab6b137120172fd6eaab1e6e0"
server_key_version: "encryptionkey@0"
I20260927 22:01:22.672011 6788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:22.693344 6788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:22.698884 6788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:22.699218 6788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:22.723881 6788 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:46295
I20260927 22:01:22.723994 6854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:46295 every 8 connection(s)
I20260927 22:01:22.725615 6788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:22.731316 6855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:22.735575 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 6788
I20260927 22:01:22.735881 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:22.736555 3708 external_mini_cluster.cc:1428] Setting key 808f4c77f8cd08249d386912aab3b628
I20260927 22:01:22.748377 6855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e: Bootstrap starting.
I20260927 22:01:22.753569 6855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:22.755390 6855 log.cc:824] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:01:22 dist-test-slave-sg4j krb5kdc[6775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:22.759676 6855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e: No bootstrap required, opened a new log
I20260927 22:01:22.767223 6858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:22.739291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.3.159.62:46295)
0927 22:01:22.740253 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:22.740267 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:22.740318 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:01:22.743107 (+ 2789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:22.743141 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:22.743715 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:22.744145 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:22.746509 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:22.748398 (+ 1889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:22.750560 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:22.751496 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:22.751730 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:22.757859 (+ 6129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:22.757870 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:22.757892 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:22.757937 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:22.761702 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:22.762570 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:22.762578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:22.762587 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:22.762794 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:22.763248 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:22.763252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:22.763254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:22.763887 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:22.764271 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:22.764772 (+ 501us) server_negotiation.cc:299] Negotiation successful
0927 22:01:22.765264 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":463,"threads_started":1}
I20260927 22:01:22.769626 6855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df232b4bae48dda942df172351830e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46295 } }
I20260927 22:01:22.770149 6855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:22.770293 6855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17df232b4bae48dda942df172351830e, State: Initialized, Role: FOLLOWER
I20260927 22:01:22.771123 6855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df232b4bae48dda942df172351830e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46295 } }
I20260927 22:01:22.771427 6855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:22.771586 6855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:22.771782 6855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:22.774914 6855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df232b4bae48dda942df172351830e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46295 } }
I20260927 22:01:22.775700 6855 leader_election.cc:302] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17df232b4bae48dda942df172351830e; no voters:
I20260927 22:01:22.776666 6855 leader_election.cc:288] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:22.776883 6860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:22.778235 6860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [term 1 LEADER]: Becoming Leader. State: Replica: 17df232b4bae48dda942df172351830e, State: Running, Role: LEADER
I20260927 22:01:22.779052 6860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df232b4bae48dda942df172351830e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46295 } }
I20260927 22:01:22.779944 6855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:22.782387 6861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17df232b4bae48dda942df172351830e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df232b4bae48dda942df172351830e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46295 } } }
I20260927 22:01:22.782775 6862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17df232b4bae48dda942df172351830e. Latest consensus state: current_term: 1 leader_uuid: "17df232b4bae48dda942df172351830e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17df232b4bae48dda942df172351830e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46295 } } }
I20260927 22:01:22.783329 6861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:22.783324 6862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:22.788491 6869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:22.795735 6869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:22.808179 6869 catalog_manager.cc:1347] Generated new cluster ID: 6d4e40e7726b4dcbb3471dd038433037
I20260927 22:01:22.808357 6869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:22.827466 6869 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:22.829324 6869 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:22.841815 6869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17df232b4bae48dda942df172351830e: Generated new TSK 0
I20260927 22:01:22.843047 6869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:01:22.914765 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:46295
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:23.356979 6881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:23.357442 6881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:23.357579 6881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:23.366138 6881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:23.366312 6881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:23.366446 6881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:01:23.378353 6881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:23.380198 6881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:23.382059 6881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:23.399091 6886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:23.399164 6887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:23.399514 6889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:23.400573 6881 server_base.cc:1034] running on GCE node
I20260927 22:01:23.401466 6881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:23.403005 6881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:23.404256 6881 hybrid_clock.cc:648] HybridClock initialized: now 1790546483404150 us; error 98 us; skew 500 ppm
I20260927 22:01:23.404701 6881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:23 dist-test-slave-sg4j krb5kdc[6775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546483, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:23.408869 6881 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:01:23.410754 6881 webserver.cc:460] Webserver started at http://127.3.159.1:35673/ using document root <none> and password file <none>
I20260927 22:01:23.411535 6881 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:23.411645 6881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:23.411912 6881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:23.415164 6881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "573269d74f9d4024b2e4cf04ea1e8570"
format_stamp: "Formatted at 2026-09-27 22:01:23 on dist-test-slave-sg4j"
server_key: "46d1fd1b09d3fc9cfb1f34c6844cea4c"
server_key_iv: "b945f70414360d8ad51593a785f70e97"
server_key_version: "encryptionkey@0"
I20260927 22:01:23.416034 6881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "573269d74f9d4024b2e4cf04ea1e8570"
format_stamp: "Formatted at 2026-09-27 22:01:23 on dist-test-slave-sg4j"
server_key: "46d1fd1b09d3fc9cfb1f34c6844cea4c"
server_key_iv: "b945f70414360d8ad51593a785f70e97"
server_key_version: "encryptionkey@0"
I20260927 22:01:23.422624 6881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 22:01:23.428570 6896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:23.430881 6881 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:01:23.431178 6881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "573269d74f9d4024b2e4cf04ea1e8570"
format_stamp: "Formatted at 2026-09-27 22:01:23 on dist-test-slave-sg4j"
server_key: "46d1fd1b09d3fc9cfb1f34c6844cea4c"
server_key_iv: "b945f70414360d8ad51593a785f70e97"
server_key_version: "encryptionkey@0"
I20260927 22:01:23.431428 6881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:23.467332 6881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:23.472337 6881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:23.472681 6881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:23.473904 6881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:23.475651 6881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:23.475750 6881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:23.475924 6881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:23.476009 6881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:23.518013 6881 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:38361
I20260927 22:01:23.518131 7008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:38361 every 8 connection(s)
I20260927 22:01:23.519518 6881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:01:23.526005 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 6881
I20260927 22:01:23.526273 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:01:23.527006 3708 external_mini_cluster.cc:1428] Setting key 6cfbd73123f9d6b6d1351eecae66c066
Sep 27 22:01:23 dist-test-slave-sg4j krb5kdc[6775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546483, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:23.542722 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:23.523098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:54487 (local address 127.3.159.62:46295)
0927 22:01:23.523614 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:23.523619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:23.524996 (+ 1377us) server_negotiation.cc:407] Connection header received
0927 22:01:23.527376 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:23.527383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:23.527483 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:23.527622 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:23.529760 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:23.530489 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:23.532160 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:23.532548 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:23.532696 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:23.536438 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:23.536444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:23.536447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:23.536482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:23.539313 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:23.540178 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:23.540183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:23.540185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:23.540304 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:23.540771 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:23.540783 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:23.540785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:23.541049 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:23.541206 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:23.542112 (+ 906us) server_negotiation.cc:299] Negotiation successful
0927 22:01:23.542345 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":306,"threads_started":1}
I20260927 22:01:23.544245 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:23.523757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:46295 (local address 127.3.159.1:54487)
0927 22:01:23.524774 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:23.524836 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:23.526959 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:23.527929 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:23.527939 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:23.528711 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:23.529553 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:23.529576 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:23.530661 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:23.530669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:23.531988 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:23.532001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:23.532799 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:23.532807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:23.533062 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:23.533832 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:23.533855 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:23.536156 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:23.539497 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:23.539508 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:23.539529 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:23.540010 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:23.540423 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:23.540428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:23.540432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:23.540646 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:23.541182 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:23.541191 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:23.541698 (+ 507us) client_negotiation.cc:769] Sending connection context
0927 22:01:23.542011 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 22:01:23.542616 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":415,"threads_started":1}
I20260927 22:01:23.547219 7009 heartbeater.cc:347] Connected to a master server at 127.3.159.62:46295
I20260927 22:01:23.547804 7009 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:23.549225 7009 heartbeater.cc:511] Master 127.3.159.62:46295 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:01:23.552722 6820 ts_manager.cc:194] Registered new tserver with Master: 573269d74f9d4024b2e4cf04ea1e8570 (127.3.159.1:38361)
I20260927 22:01:23.554469 6820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:54487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:01:23.586707 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:46295
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:23.979513 7015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:23.979902 7015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:23.980005 7015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:23.987637 7015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:23.987782 7015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:23.988011 7015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:01:23.999925 7015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:24.001653 7015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:24.003124 7015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:24.019043 7021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:24.019439 7020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:24.019309 7023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:24.021023 7015 server_base.cc:1034] running on GCE node
I20260927 22:01:24.022002 7015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:24.023322 7015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:24.024649 7015 hybrid_clock.cc:648] HybridClock initialized: now 1790546484024580 us; error 68 us; skew 500 ppm
I20260927 22:01:24.025143 7015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:24 dist-test-slave-sg4j krb5kdc[6775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546484, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:24.029137 7015 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:01:24.031235 7015 webserver.cc:460] Webserver started at http://127.3.159.2:44217/ using document root <none> and password file <none>
I20260927 22:01:24.032114 7015 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:24.032238 7015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:24.032555 7015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:24.036080 7015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "a91232bdc6c34e8494ec77f3db64ff29"
format_stamp: "Formatted at 2026-09-27 22:01:24 on dist-test-slave-sg4j"
server_key: "466fd189faa9a8f4a34342a83f353a1b"
server_key_iv: "985054b1cd5c6ce76619b422d3da14c7"
server_key_version: "encryptionkey@0"
I20260927 22:01:24.037086 7015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "a91232bdc6c34e8494ec77f3db64ff29"
format_stamp: "Formatted at 2026-09-27 22:01:24 on dist-test-slave-sg4j"
server_key: "466fd189faa9a8f4a34342a83f353a1b"
server_key_iv: "985054b1cd5c6ce76619b422d3da14c7"
server_key_version: "encryptionkey@0"
I20260927 22:01:24.044260 7015 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s
I20260927 22:01:24.049839 7030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:24.051891 7015 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 22:01:24.052181 7015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "a91232bdc6c34e8494ec77f3db64ff29"
format_stamp: "Formatted at 2026-09-27 22:01:24 on dist-test-slave-sg4j"
server_key: "466fd189faa9a8f4a34342a83f353a1b"
server_key_iv: "985054b1cd5c6ce76619b422d3da14c7"
server_key_version: "encryptionkey@0"
I20260927 22:01:24.052438 7015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:24.070734 7015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:24.076612 7015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:24.076987 7015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:24.078405 7015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:24.080224 7015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:24.080338 7015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:24.080536 7015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:24.080611 7015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:24.123217 7015 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:43827
I20260927 22:01:24.123334 7142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:43827 every 8 connection(s)
I20260927 22:01:24.125231 7015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:01:24.132682 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 7015
I20260927 22:01:24.132899 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:01:24.133522 3708 external_mini_cluster.cc:1428] Setting key 6c45fba3d08382de89696882151f1031
Sep 27 22:01:24 dist-test-slave-sg4j krb5kdc[6775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546484, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:24.152879 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:24.128639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:59587 (local address 127.3.159.62:46295)
0927 22:01:24.129169 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:24.129174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:24.130441 (+ 1267us) server_negotiation.cc:407] Connection header received
0927 22:01:24.132606 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:24.132612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:24.132721 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:24.132887 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:24.135280 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.136293 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.138160 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.138581 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.138731 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:24.144131 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:24.144141 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:24.144147 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:24.144210 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:24.148218 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.149560 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.149570 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.149577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.149775 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.150401 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.150409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.150415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.150828 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:24.150994 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:24.152047 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0927 22:01:24.152365 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":384,"thread_start_us":258,"threads_started":1}
I20260927 22:01:24.154240 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:24.129211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:46295 (local address 127.3.159.2:59587)
0927 22:01:24.130190 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:24.130244 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:24.132252 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:24.133332 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:24.133347 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:24.134244 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:24.135052 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:24.135074 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.136459 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.136465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:24.137948 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:24.137968 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.138838 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.138845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:24.139111 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:24.140125 (+ 1014us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:24.140165 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:24.143625 (+ 3460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:24.148517 (+ 4892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:24.148530 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:24.148552 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:24.149288 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:24.149944 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:24.149949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:24.149952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:24.150233 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:24.150981 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:24.150990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:24.151525 (+ 535us) client_negotiation.cc:769] Sending connection context
0927 22:01:24.151883 (+ 358us) client_negotiation.cc:240] Negotiation successful
0927 22:01:24.152463 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":420,"threads_started":1}
I20260927 22:01:24.156394 7143 heartbeater.cc:347] Connected to a master server at 127.3.159.62:46295
I20260927 22:01:24.156797 7143 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
I20260927 22:01:24.158068 7143 heartbeater.cc:511] Master 127.3.159.62:46295 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
I20260927 22:01:24.160547 6820 ts_manager.cc:194] Registered new tserver with Master: a91232bdc6c34e8494ec77f3db64ff29 (127.3.159.2:43827)
I20260927 22:01:24.161623 6820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:59587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:01:24.190327 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:46295
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:24.558602 7009 heartbeater.cc:503] Master 127.3.159.62:46295 was elected leader, sending a full tablet report...
W20260927 22:01:24.576674 7149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:24.577049 7149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:24.577176 7149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:24.585812 7149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:24.585975 7149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:24.586097 7149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:01:24.597841 7149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:46295
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:24.599601 7149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:24.601220 7149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:24.618376 7157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:24.618311 7155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:24.618496 7154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:24.619731 7149 server_base.cc:1034] running on GCE node
I20260927 22:01:24.620675 7149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:24.622143 7149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:24.623445 7149 hybrid_clock.cc:648] HybridClock initialized: now 1790546484623332 us; error 108 us; skew 500 ppm
I20260927 22:01:24.623857 7149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:24 dist-test-slave-sg4j krb5kdc[6775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546484, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:24.627964 7149 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:01:24.629992 7149 webserver.cc:460] Webserver started at http://127.3.159.3:36541/ using document root <none> and password file <none>
I20260927 22:01:24.630916 7149 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:24.631042 7149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:24.631417 7149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:24.635146 7149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "2cfc0b83fbfe4c0790b48a33bd19c88b"
format_stamp: "Formatted at 2026-09-27 22:01:24 on dist-test-slave-sg4j"
server_key: "39bfe081369576e394857dd72f96f37f"
server_key_iv: "8aba7a641fb556166a4c2f5c28e878e1"
server_key_version: "encryptionkey@0"
I20260927 22:01:24.636124 7149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "2cfc0b83fbfe4c0790b48a33bd19c88b"
format_stamp: "Formatted at 2026-09-27 22:01:24 on dist-test-slave-sg4j"
server_key: "39bfe081369576e394857dd72f96f37f"
server_key_iv: "8aba7a641fb556166a4c2f5c28e878e1"
server_key_version: "encryptionkey@0"
I20260927 22:01:24.643457 7149 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260927 22:01:24.648061 7164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:24.650411 7149 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 22:01:24.650671 7149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "2cfc0b83fbfe4c0790b48a33bd19c88b"
format_stamp: "Formatted at 2026-09-27 22:01:24 on dist-test-slave-sg4j"
server_key: "39bfe081369576e394857dd72f96f37f"
server_key_iv: "8aba7a641fb556166a4c2f5c28e878e1"
server_key_version: "encryptionkey@0"
I20260927 22:01:24.650950 7149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:24.669564 7149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:24.675948 7149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:24.676416 7149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:24.677937 7149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:24.679900 7149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:24.680009 7149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:24.680171 7149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:24.680254 7149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:24.726389 7149 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:43085
I20260927 22:01:24.726433 7276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:43085 every 8 connection(s)
I20260927 22:01:24.728359 7149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:01:24.737272 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 7149
I20260927 22:01:24.737567 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:01:24.738042 3708 external_mini_cluster.cc:1428] Setting key 1395caab1cbf5cc9beaf57fd05bcd955
Sep 27 22:01:24 dist-test-slave-sg4j krb5kdc[6775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546484, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:24.757290 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:24.731774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:35681 (local address 127.3.159.62:46295)
0927 22:01:24.732415 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:24.732422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:24.733775 (+ 1353us) server_negotiation.cc:407] Connection header received
0927 22:01:24.736231 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:24.736236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:24.736324 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:24.736476 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:24.739499 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.740647 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.742933 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.743395 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.743579 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:24.748298 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:24.748305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:24.748309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:24.748355 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:24.751661 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.753008 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.753014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.753017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.753287 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.754303 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.754310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.754313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.754712 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:24.754977 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:24.756508 (+ 1531us) server_negotiation.cc:299] Negotiation successful
0927 22:01:24.756863 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":304,"threads_started":1}
I20260927 22:01:24.758616 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:24.732301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:46295 (local address 127.3.159.3:35681)
0927 22:01:24.733485 (+ 1184us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:24.733560 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:24.735688 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:24.736899 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:24.736912 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:24.738140 (+ 1228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:24.739226 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:24.739257 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.740879 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.740885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:24.742739 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:24.742758 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.743717 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.743725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:24.744034 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:24.744934 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:24.744965 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:24.747921 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:24.751986 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:24.751998 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:24.752021 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:24.752690 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:24.753577 (+ 887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:24.753583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:24.753591 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:24.754084 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:24.754986 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:24.754996 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:24.755885 (+ 889us) client_negotiation.cc:769] Sending connection context
0927 22:01:24.756278 (+ 393us) client_negotiation.cc:240] Negotiation successful
0927 22:01:24.756980 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":391,"threads_started":1}
I20260927 22:01:24.760789 7277 heartbeater.cc:347] Connected to a master server at 127.3.159.62:46295
I20260927 22:01:24.761170 7277 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:24.762154 7277 heartbeater.cc:511] Master 127.3.159.62:46295 requested a full tablet report, sending...
I20260927 22:01:24.763988 6820 ts_manager.cc:194] Registered new tserver with Master: 2cfc0b83fbfe4c0790b48a33bd19c88b (127.3.159.3:43085)
I20260927 22:01:24.764920 6820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:35681
I20260927 22:01:24.769284 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:01:24 dist-test-slave-sg4j krb5kdc[6775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546484, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 22:01:24 dist-test-slave-sg4j krb5kdc[6775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546484, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:24.832192 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:24.815128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.3.159.62:46295)
0927 22:01:24.815358 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:24.815363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:24.815851 (+ 488us) server_negotiation.cc:407] Connection header received
0927 22:01:24.816013 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:24.816018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:24.816120 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:24.816265 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:24.818028 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.818881 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.820314 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.820784 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.820899 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:24.825117 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:24.825123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:24.825127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:24.825178 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:24.828456 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.829701 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.829707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.829709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.829885 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.830607 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.830612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.830614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.831019 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:24.831325 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:24.831668 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 22:01:24.831905 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260927 22:01:24.833259 6820 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59402
I20260927 22:01:24.835875 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 6881
Sep 27 22:01:24 dist-test-slave-sg4j krb5kdc[6775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546484, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:01:24.858170 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:24.837940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.3.159.1:38361)
0927 22:01:24.839016 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:24.839025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:24.839056 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:01:24.839126 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:24.839134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:24.839432 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:24.839633 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:24.841067 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.842302 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.843701 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.844404 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.844563 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:24.849833 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:24.849852 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:24.849873 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:24.849924 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:24.853629 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.854649 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.854660 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.854662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.854908 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.855646 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.855651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.855654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.856378 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:24.856734 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:24.857199 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 22:01:24.857727 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":744,"threads_started":1}
W20260927 22:01:24.859620 7004 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39252
W20260927 22:01:24.860980 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 6881: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:01:24.861289 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 6881
I20260927 22:01:24.881258 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 7015
Sep 27 22:01:24 dist-test-slave-sg4j krb5kdc[6775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546484, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:01:24.901568 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:24.882550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.3.159.2:43827)
0927 22:01:24.883135 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:24.883144 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:24.883189 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:01:24.883270 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:24.883279 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:24.883608 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:24.883773 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:24.885503 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.886577 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.888108 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.888749 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.888951 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:24.894110 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:24.894125 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:24.894154 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:24.894217 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:24.897744 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.898724 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.898735 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.898740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.898922 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.899443 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.899448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.899450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.899903 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:24.900207 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:24.900627 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 22:01:24.900987 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":273,"threads_started":1}
W20260927 22:01:24.903237 7138 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55256
W20260927 22:01:24.904628 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 7015: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:01:24.904903 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 7015
I20260927 22:01:24.923940 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 7149
Sep 27 22:01:24 dist-test-slave-sg4j krb5kdc[6775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546484, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:01:24.943374 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:24.925048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.3.159.3:43085)
0927 22:01:24.925892 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:24.925901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:24.925938 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:01:24.926010 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:24.926019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:24.926419 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:24.926643 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:24.928198 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.929455 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.930818 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:24.931471 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:24.931627 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:24.936645 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:24.936662 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:24.936683 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:24.936729 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:24.940012 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.940801 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.940811 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.940813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.940945 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:24.941354 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:24.941366 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:24.941369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:24.941863 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:24.942242 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:24.942580 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 22:01:24.942912 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":454,"threads_started":1}
W20260927 22:01:24.944996 7272 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39330
W20260927 22:01:24.946434 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 7149: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:01:24.946645 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 7149
I20260927 22:01:24.966212 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 6788
I20260927 22:01:24.967741 6850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:25.085194 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 6788
2026-09-27T22:01:25Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4289 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:01:25 dist-test-slave-sg4j krb5kdc[7295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:01:25 dist-test-slave-sg4j krb5kdc[7295](info): set up 2 sockets
Sep 27 22:01:25 dist-test-slave-sg4j krb5kdc[7295](info): commencing operation
krb5kdc: starting...
W20260927 22:01:26.230417 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.008s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:01:26 dist-test-slave-sg4j krb5kdc[7295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:01:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:01:26.467692 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:40877
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:40877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:26.868862 7308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:26.869298 7308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:26.869513 7308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:26.878306 7308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:26.878501 7308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:26.878557 7308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:26.878696 7308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:26.878763 7308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:01:26.891312 7308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:40877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:40877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:26.893246 7308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:26.895044 7308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:26.916875 7316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:26.917116 7313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:26.917634 7314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:26.918699 7308 server_base.cc:1034] running on GCE node
I20260927 22:01:26.919958 7308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:26.922662 7308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:26.924150 7308 hybrid_clock.cc:648] HybridClock initialized: now 1790546486924036 us; error 115 us; skew 500 ppm
I20260927 22:01:26.924679 7308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:26 dist-test-slave-sg4j krb5kdc[7295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546486, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:26.929121 7308 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:01:26.931622 7308 webserver.cc:460] Webserver started at http://127.3.159.62:40695/ using document root <none> and password file <none>
I20260927 22:01:26.932736 7308 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:26.932883 7308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:26.933303 7308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:26.937577 7308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "2e006d2d3e0140039e054a5b0eecec3c"
format_stamp: "Formatted at 2026-09-27 22:01:26 on dist-test-slave-sg4j"
server_key: "02fe0180872f4e27a7d325bcebd78c48"
server_key_iv: "2896871d3aa7e25b39119695219b7688"
server_key_version: "encryptionkey@0"
I20260927 22:01:26.938759 7308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "2e006d2d3e0140039e054a5b0eecec3c"
format_stamp: "Formatted at 2026-09-27 22:01:26 on dist-test-slave-sg4j"
server_key: "02fe0180872f4e27a7d325bcebd78c48"
server_key_iv: "2896871d3aa7e25b39119695219b7688"
server_key_version: "encryptionkey@0"
I20260927 22:01:26.946388 7308 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:01:26.951687 7323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:26.954211 7308 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:01:26.954540 7308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "2e006d2d3e0140039e054a5b0eecec3c"
format_stamp: "Formatted at 2026-09-27 22:01:26 on dist-test-slave-sg4j"
server_key: "02fe0180872f4e27a7d325bcebd78c48"
server_key_iv: "2896871d3aa7e25b39119695219b7688"
server_key_version: "encryptionkey@0"
I20260927 22:01:26.954835 7308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:26.977773 7308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:26.984320 7308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:26.984743 7308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:27.010668 7308 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:40877
I20260927 22:01:27.010730 7374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:40877 every 8 connection(s)
I20260927 22:01:27.012809 7308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:27.015090 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 7308
I20260927 22:01:27.015359 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:27.016331 3708 external_mini_cluster.cc:1428] Setting key 28d42baaad05640d8df90f96c1fda662
I20260927 22:01:27.018966 7375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:27.035454 7375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c: Bootstrap starting.
Sep 27 22:01:27 dist-test-slave-sg4j krb5kdc[7295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:27.041046 7375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:27.043323 7375 log.cc:824] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:27.048591 7375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c: No bootstrap required, opened a new log
I20260927 22:01:27.049286 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:27.019183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.3.159.62:40877)
0927 22:01:27.020543 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:27.020566 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:27.020628 (+ 62us) server_negotiation.cc:407] Connection header received
0927 22:01:27.022764 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:27.022799 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:27.023614 (+ 815us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:27.024277 (+ 663us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:27.026570 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:27.028439 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:27.030373 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:27.031473 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:27.031814 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:27.036913 (+ 5099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:27.036926 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:27.036952 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:27.037026 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:27.041773 (+ 4747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:27.043217 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:27.043228 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:27.043238 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:27.043697 (+ 459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:27.044538 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:27.044544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:27.044547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:27.045508 (+ 961us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:27.045923 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:27.046784 (+ 861us) server_negotiation.cc:299] Negotiation successful
0927 22:01:27.047687 (+ 903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1052,"thread_start_us":705,"threads_started":1}
I20260927 22:01:27.058903 7375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e006d2d3e0140039e054a5b0eecec3c" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40877 } }
I20260927 22:01:27.059705 7375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:27.059863 7375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e006d2d3e0140039e054a5b0eecec3c, State: Initialized, Role: FOLLOWER
I20260927 22:01:27.060853 7375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e006d2d3e0140039e054a5b0eecec3c" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40877 } }
I20260927 22:01:27.061188 7375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:27.061300 7375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:27.061573 7375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:27.064515 7375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e006d2d3e0140039e054a5b0eecec3c" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40877 } }
I20260927 22:01:27.065100 7375 leader_election.cc:302] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e006d2d3e0140039e054a5b0eecec3c; no voters:
I20260927 22:01:27.065830 7375 leader_election.cc:288] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:27.066123 7380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:27.067113 7380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [term 1 LEADER]: Becoming Leader. State: Replica: 2e006d2d3e0140039e054a5b0eecec3c, State: Running, Role: LEADER
I20260927 22:01:27.067945 7380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e006d2d3e0140039e054a5b0eecec3c" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40877 } }
I20260927 22:01:27.068682 7375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:27.072955 7381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e006d2d3e0140039e054a5b0eecec3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e006d2d3e0140039e054a5b0eecec3c" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40877 } } }
I20260927 22:01:27.073930 7381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:27.073035 7382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e006d2d3e0140039e054a5b0eecec3c. Latest consensus state: current_term: 1 leader_uuid: "2e006d2d3e0140039e054a5b0eecec3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e006d2d3e0140039e054a5b0eecec3c" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40877 } } }
I20260927 22:01:27.075250 7382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:27.079452 7387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:27.087613 7387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:27.100659 7387 catalog_manager.cc:1347] Generated new cluster ID: 074a0cdc5c49493fbdf8655329ebfb0d
I20260927 22:01:27.100873 7387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:27.109968 7387 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:27.111625 7387 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:27.128630 7387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e006d2d3e0140039e054a5b0eecec3c: Generated new TSK 0
I20260927 22:01:27.130311 7387 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:01:27.183665 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40877
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:27.587715 7401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:27.588073 7401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:27.588241 7401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:27.596788 7401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:27.596937 7401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:27.597150 7401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:01:27.607936 7401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:27.609561 7401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:27.611095 7401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:27.627324 7406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:27.627514 7407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:27.627310 7409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:27.628659 7401 server_base.cc:1034] running on GCE node
I20260927 22:01:27.629465 7401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:27.630627 7401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:27.631929 7401 hybrid_clock.cc:648] HybridClock initialized: now 1790546487631840 us; error 82 us; skew 500 ppm
I20260927 22:01:27.632483 7401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:27 dist-test-slave-sg4j krb5kdc[7295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546487, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:27.636631 7401 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:01:27.638556 7401 webserver.cc:460] Webserver started at http://127.3.159.1:37331/ using document root <none> and password file <none>
I20260927 22:01:27.639369 7401 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:27.639537 7401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:27.639880 7401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:27.643180 7401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "44bbc9f9acd84abd935a05bc554dd40d"
format_stamp: "Formatted at 2026-09-27 22:01:27 on dist-test-slave-sg4j"
server_key: "711010cb9df982e1156aa22e36bb5f26"
server_key_iv: "5f70aef058e0590d86d587d53cc7bb44"
server_key_version: "encryptionkey@0"
I20260927 22:01:27.644066 7401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "44bbc9f9acd84abd935a05bc554dd40d"
format_stamp: "Formatted at 2026-09-27 22:01:27 on dist-test-slave-sg4j"
server_key: "711010cb9df982e1156aa22e36bb5f26"
server_key_iv: "5f70aef058e0590d86d587d53cc7bb44"
server_key_version: "encryptionkey@0"
I20260927 22:01:27.650589 7401 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 22:01:27.655654 7416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:27.657924 7401 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260927 22:01:27.658190 7401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "44bbc9f9acd84abd935a05bc554dd40d"
format_stamp: "Formatted at 2026-09-27 22:01:27 on dist-test-slave-sg4j"
server_key: "711010cb9df982e1156aa22e36bb5f26"
server_key_iv: "5f70aef058e0590d86d587d53cc7bb44"
server_key_version: "encryptionkey@0"
I20260927 22:01:27.658450 7401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:27.681536 7401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:27.687413 7401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:27.687819 7401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:27.689263 7401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:27.691237 7401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:27.691385 7401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:27.691567 7401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:27.691654 7401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:27.739037 7401 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:34857
I20260927 22:01:27.739073 7528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:34857 every 8 connection(s)
I20260927 22:01:27.740965 7401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:01:27.749747 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 7401
I20260927 22:01:27.750010 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:01:27.750777 3708 external_mini_cluster.cc:1428] Setting key 5b3a3ae1b7d3a8cb3f4088041c91750c
Sep 27 22:01:27 dist-test-slave-sg4j krb5kdc[7295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546487, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:27.769101 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:27.744660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:55095 (local address 127.3.159.62:40877)
0927 22:01:27.745361 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:27.745369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:27.746912 (+ 1543us) server_negotiation.cc:407] Connection header received
0927 22:01:27.748967 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:27.748973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:27.749130 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:27.749373 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:27.751483 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:27.752330 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:27.754401 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:27.755275 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:27.755496 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:27.760854 (+ 5358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:27.760861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:27.760866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:27.760915 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:27.764435 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:27.765799 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:27.765805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:27.765808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:27.765989 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:27.766671 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:27.766677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:27.766680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:27.767034 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:27.767332 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:27.768425 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0927 22:01:27.768694 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":360,"threads_started":1}
I20260927 22:01:27.770653 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:27.745544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:40877 (local address 127.3.159.1:55095)
0927 22:01:27.746678 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:27.746743 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:27.748588 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:27.749754 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:27.749765 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:27.750497 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:27.751232 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:27.751258 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:27.752539 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:27.752545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:27.754188 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:27.754203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:27.755616 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:27.755622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:27.755874 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:27.756845 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:27.756879 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:27.760433 (+ 3554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:27.764676 (+ 4243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:27.764690 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:27.764712 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:27.765575 (+ 863us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:27.766143 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:27.766150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:27.766157 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:27.766465 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:27.767194 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:27.767206 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:27.767803 (+ 597us) client_negotiation.cc:769] Sending connection context
0927 22:01:27.768162 (+ 359us) client_negotiation.cc:240] Negotiation successful
0927 22:01:27.768883 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":417,"threads_started":1}
I20260927 22:01:27.773116 7529 heartbeater.cc:347] Connected to a master server at 127.3.159.62:40877
I20260927 22:01:27.773708 7529 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:27.774750 7529 heartbeater.cc:511] Master 127.3.159.62:40877 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:01:27.777809 7340 ts_manager.cc:194] Registered new tserver with Master: 44bbc9f9acd84abd935a05bc554dd40d (127.3.159.1:34857)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:01:27.780217 7340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:55095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:01:27.810990 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40877
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:28.234721 7535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:28.235126 7535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:28.235235 7535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:28.244000 7535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:28.244175 7535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:28.244319 7535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:01:28.256152 7535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:28.258208 7535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:28.260512 7535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:28.281600 7541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:28.281560 7543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:28.281847 7540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:28.283340 7535 server_base.cc:1034] running on GCE node
I20260927 22:01:28.284385 7535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:28.286036 7535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:28.287405 7535 hybrid_clock.cc:648] HybridClock initialized: now 1790546488287314 us; error 87 us; skew 500 ppm
I20260927 22:01:28.287993 7535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:28 dist-test-slave-sg4j krb5kdc[7295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546488, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:28.292935 7535 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:01:28.295475 7535 webserver.cc:460] Webserver started at http://127.3.159.2:35425/ using document root <none> and password file <none>
I20260927 22:01:28.296550 7535 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:28.296697 7535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:28.297199 7535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:28.301761 7535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "3a5d19bcd4184f0b8ae4afc597165be1"
format_stamp: "Formatted at 2026-09-27 22:01:28 on dist-test-slave-sg4j"
server_key: "49cf6bf10e1fea8be8457224a35294dc"
server_key_iv: "aba51a34d1be1a617b12a2ed519c1ad9"
server_key_version: "encryptionkey@0"
I20260927 22:01:28.302855 7535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "3a5d19bcd4184f0b8ae4afc597165be1"
format_stamp: "Formatted at 2026-09-27 22:01:28 on dist-test-slave-sg4j"
server_key: "49cf6bf10e1fea8be8457224a35294dc"
server_key_iv: "aba51a34d1be1a617b12a2ed519c1ad9"
server_key_version: "encryptionkey@0"
I20260927 22:01:28.311254 7535 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260927 22:01:28.316649 7550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:28.318882 7535 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 22:01:28.319121 7535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "3a5d19bcd4184f0b8ae4afc597165be1"
format_stamp: "Formatted at 2026-09-27 22:01:28 on dist-test-slave-sg4j"
server_key: "49cf6bf10e1fea8be8457224a35294dc"
server_key_iv: "aba51a34d1be1a617b12a2ed519c1ad9"
server_key_version: "encryptionkey@0"
I20260927 22:01:28.319435 7535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:28.350219 7535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:28.356537 7535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:28.356981 7535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:28.358887 7535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:28.361470 7535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:28.361584 7535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 22:01:28.361696 7535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:28.361745 7535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:28.407455 7535 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:34741
I20260927 22:01:28.407514 7662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:34741 every 8 connection(s)
I20260927 22:01:28.409176 7535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:01:28.410390 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 7535
I20260927 22:01:28.410629 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:01:28.411180 3708 external_mini_cluster.cc:1428] Setting key 63e541db2435c0a1c26f580e8978bef6
Sep 27 22:01:28 dist-test-slave-sg4j krb5kdc[7295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546488, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
I20260927 22:01:28.439805 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:28.412982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:51217 (local address 127.3.159.62:40877)
0927 22:01:28.413634 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:28.413641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:28.415400 (+ 1759us) server_negotiation.cc:407] Connection header received
0927 22:01:28.418018 (+ 2618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:28.418028 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:28.418147 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:28.418332 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:28.420715 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:28.421771 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:28.423802 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:28.424382 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:28.424540 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:28.430246 (+ 5706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:28.430254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:28.430259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:28.430318 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:28.434415 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:28.435781 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:28.435787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:28.435790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:28.435983 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:28.436908 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:28.436914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:28.436923 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:28.437623 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:28.437829 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:28.439008 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0927 22:01:28.439353 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":349,"threads_started":1}
I20260927 22:01:28.441452 7666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:28.413967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:40877 (local address 127.3.159.2:51217)
0927 22:01:28.415118 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:28.415206 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:28.417534 (+ 2328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:28.418671 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:28.418683 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:28.419473 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:28.420373 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:28.420401 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:28.421993 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:28.421998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:28.423570 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:28.423595 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:28.424697 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:28.424706 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:28.425087 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:28.426127 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:28.426159 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:28.429698 (+ 3539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:28.434722 (+ 5024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:28.434734 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:28.434753 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:28.435496 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:28.436189 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:28.436202 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:28.436205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:28.436699 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:28.437841 (+ 1142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:28.437856 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:28.438505 (+ 649us) client_negotiation.cc:769] Sending connection context
0927 22:01:28.438911 (+ 406us) client_negotiation.cc:240] Negotiation successful
0927 22:01:28.439680 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":478,"threads_started":1}
I20260927 22:01:28.444620 7663 heartbeater.cc:347] Connected to a master server at 127.3.159.62:40877
I20260927 22:01:28.445128 7663 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:28.446233 7663 heartbeater.cc:511] Master 127.3.159.62:40877 requested a full tablet report, sending...
I20260927 22:01:28.448568 7340 ts_manager.cc:194] Registered new tserver with Master: 3a5d19bcd4184f0b8ae4afc597165be1 (127.3.159.2:34741)
I20260927 22:01:28.449990 7340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:51217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:01:28.470867 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40877
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:28.785131 7529 heartbeater.cc:503] Master 127.3.159.62:40877 was elected leader, sending a full tablet report...
W20260927 22:01:28.897492 7669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:28.897964 7669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:28.898155 7669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:28.907753 7669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:28.908001 7669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:28.908270 7669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:01:28.921360 7669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:28.923353 7669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:28.925273 7669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:28.943610 7674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:28.943552 7675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:28.943576 7677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:28.945925 7669 server_base.cc:1034] running on GCE node
I20260927 22:01:28.947028 7669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:28.948705 7669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:28.950121 7669 hybrid_clock.cc:648] HybridClock initialized: now 1790546488950032 us; error 93 us; skew 500 ppm
I20260927 22:01:28.950809 7669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:28 dist-test-slave-sg4j krb5kdc[7295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546488, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:28.955816 7669 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:01:28.958274 7669 webserver.cc:460] Webserver started at http://127.3.159.3:45945/ using document root <none> and password file <none>
I20260927 22:01:28.959337 7669 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:28.959542 7669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:28.959977 7669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:28.965003 7669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "7e8620c723554a7db1f41793c841b9a5"
format_stamp: "Formatted at 2026-09-27 22:01:28 on dist-test-slave-sg4j"
server_key: "9a80c11a9bd879366a23a81ad233ea66"
server_key_iv: "452ddb2b5fad32da84b00c3ee1ce52d5"
server_key_version: "encryptionkey@0"
I20260927 22:01:28.966341 7669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "7e8620c723554a7db1f41793c841b9a5"
format_stamp: "Formatted at 2026-09-27 22:01:28 on dist-test-slave-sg4j"
server_key: "9a80c11a9bd879366a23a81ad233ea66"
server_key_iv: "452ddb2b5fad32da84b00c3ee1ce52d5"
server_key_version: "encryptionkey@0"
I20260927 22:01:28.974390 7669 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:01:28.979885 7684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:28.982180 7669 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:01:28.982563 7669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "7e8620c723554a7db1f41793c841b9a5"
format_stamp: "Formatted at 2026-09-27 22:01:28 on dist-test-slave-sg4j"
server_key: "9a80c11a9bd879366a23a81ad233ea66"
server_key_iv: "452ddb2b5fad32da84b00c3ee1ce52d5"
server_key_version: "encryptionkey@0"
I20260927 22:01:28.982903 7669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:29.016908 7669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:29.023661 7669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:29.024108 7669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:29.025872 7669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:29.027853 7669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:29.027999 7669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:29.028204 7669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:29.028290 7669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:29.075183 7669 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:44441
I20260927 22:01:29.075184 7796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:44441 every 8 connection(s)
I20260927 22:01:29.076983 7669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:01:29.081071 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 7669
I20260927 22:01:29.081243 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:01:29.081872 3708 external_mini_cluster.cc:1428] Setting key b0aaeb30b1f2531c40098230f819c04c
Sep 27 22:01:29 dist-test-slave-sg4j krb5kdc[7295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546488, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:29.107945 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:29.081669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:43985 (local address 127.3.159.62:40877)
0927 22:01:29.082370 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:29.082377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:29.083729 (+ 1352us) server_negotiation.cc:407] Connection header received
0927 22:01:29.086568 (+ 2839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:29.086573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:29.086701 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:29.086844 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:29.089707 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.090646 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.093345 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.094032 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.094225 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:29.099464 (+ 5239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:29.099472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:29.099477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:29.099540 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:29.103417 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.104702 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.104708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.104711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.104885 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.105527 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.105531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.105533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.105906 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:29.106062 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:29.107276 (+ 1214us) server_negotiation.cc:299] Negotiation successful
0927 22:01:29.107544 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":345,"threads_started":1}
I20260927 22:01:29.109483 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:29.082276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:40877 (local address 127.3.159.3:43985)
0927 22:01:29.083478 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:29.083544 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:29.085853 (+ 2309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:29.087253 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:29.087265 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:29.088316 (+ 1051us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:29.089303 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:29.089350 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.090911 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.090917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:29.092752 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:29.092779 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.094443 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.094451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:29.094803 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:29.095932 (+ 1129us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:29.095965 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:29.099030 (+ 3065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:29.103706 (+ 4676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:29.103721 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:29.103743 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:29.104494 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:29.105049 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:29.105054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:29.105057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:29.105319 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:29.106160 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:29.106170 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:29.106700 (+ 530us) client_negotiation.cc:769] Sending connection context
0927 22:01:29.107028 (+ 328us) client_negotiation.cc:240] Negotiation successful
0927 22:01:29.107554 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":497,"threads_started":1}
I20260927 22:01:29.112202 7797 heartbeater.cc:347] Connected to a master server at 127.3.159.62:40877
I20260927 22:01:29.112660 7797 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:29.113618 7797 heartbeater.cc:511] Master 127.3.159.62:40877 requested a full tablet report, sending...
I20260927 22:01:29.115535 7339 ts_manager.cc:194] Registered new tserver with Master: 7e8620c723554a7db1f41793c841b9a5 (127.3.159.3:44441)
I20260927 22:01:29.116488 7339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:43985
I20260927 22:01:29.125300 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:01:29 dist-test-slave-sg4j krb5kdc[7295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:01:29 dist-test-slave-sg4j krb5kdc[7295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:01:29.195348 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:29.171817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.3.159.1:34857)
0927 22:01:29.173763 (+ 1946us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:29.173773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:29.173815 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:01:29.173896 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:29.173905 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:29.174203 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:29.174392 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:29.176405 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.177902 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.179656 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.180546 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.180720 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:29.186499 (+ 5779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:29.186525 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:29.186548 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:29.186609 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:29.191051 (+ 4442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.192263 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.192274 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.192277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.192489 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.193050 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.193055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.193057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.193632 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:29.193944 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:29.194556 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 22:01:29.194849 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1717,"thread_start_us":1483,"threads_started":1}
Sep 27 22:01:29 dist-test-slave-sg4j krb5kdc[7295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:29.220146 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:29.201675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54318 (local address 127.3.159.62:40877)
0927 22:01:29.201993 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:29.202000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:29.202242 (+ 242us) server_negotiation.cc:407] Connection header received
0927 22:01:29.202570 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:29.202575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:29.202658 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:29.202789 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:29.204348 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.205178 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.206734 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.207211 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.207377 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:29.212717 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:29.212724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:29.212729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:29.212784 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:29.216578 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.217731 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.217737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.217741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.217901 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.218350 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.218356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.218358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.218761 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:29.219067 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:29.219406 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 22:01:29.219717 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
W20260927 22:01:29.221161 7339 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54318
I20260927 22:01:29.223291 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 7401
I20260927 22:01:29.239837 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:29.224673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.3.159.1:34857)
0927 22:01:29.224958 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:29.224968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:29.225168 (+ 200us) server_negotiation.cc:407] Connection header received
0927 22:01:29.225514 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:29.225523 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:29.225629 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:29.225782 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:29.227368 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.228227 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.229870 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.230368 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.230535 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:29.233508 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:29.233514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:29.233518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:29.233564 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:29.236747 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.237670 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.237681 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.237683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.237837 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.238260 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.238269 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.238271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.238555 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:29.238833 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:29.239161 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 22:01:29.239424 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:01:29.240945 7524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:29.336674 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 7401
I20260927 22:01:29.358286 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 7535
Sep 27 22:01:29 dist-test-slave-sg4j krb5kdc[7295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:01:29.383572 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:29.359585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.3.159.2:34741)
0927 22:01:29.360492 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:29.360502 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:29.360540 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:01:29.360626 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:29.360636 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:29.360943 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:29.361186 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:29.363036 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.364343 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.366484 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.367251 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.367574 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:29.373711 (+ 6137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:29.373732 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:29.373874 (+ 142us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:29.373951 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:29.378833 (+ 4882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.380230 (+ 1397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.380240 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.380243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.380445 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.381182 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.381187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.381189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.381733 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:29.382449 (+ 716us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:29.382797 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 22:01:29.383074 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":438,"threads_started":1}
I20260927 22:01:29.385874 7658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:29.489210 7663 heartbeater.cc:503] Master 127.3.159.62:40877 was elected leader, sending a full tablet report...
I20260927 22:01:29.490617 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 7535
W20260927 22:01:29.508903 7327 connection.cc:535] server connection from 127.3.159.2:51217 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 22:01:29.509197 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 7669
Sep 27 22:01:29 dist-test-slave-sg4j krb5kdc[7295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:01:29.532891 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:29.510855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.3.159.3:44441)
0927 22:01:29.511543 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:29.511552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:29.511593 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:01:29.511677 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:29.511685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:29.511999 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:29.512194 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:29.513866 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.515183 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.516932 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:29.517800 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:29.517984 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:29.524337 (+ 6353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:29.524361 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:29.524385 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:29.524446 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:29.528403 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.530023 (+ 1620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.530034 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.530037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.530233 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:29.530936 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:29.530940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:29.530942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:29.531428 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:29.531731 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:29.532165 (+ 434us) server_negotiation.cc:299] Negotiation successful
0927 22:01:29.532450 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":409,"threads_started":1}
I20260927 22:01:29.534860 7792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:29.631513 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 7669
I20260927 22:01:29.650125 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 7308
I20260927 22:01:29.651799 7370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:29.773629 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 7308
2026-09-27T22:01:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4689 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:01:29 dist-test-slave-sg4j krb5kdc[7817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:01:29 dist-test-slave-sg4j krb5kdc[7817](info): set up 2 sockets
Sep 27 22:01:29 dist-test-slave-sg4j krb5kdc[7817](info): commencing operation
krb5kdc: starting...
W20260927 22:01:30.920617 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:01:31 dist-test-slave-sg4j krb5kdc[7817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:01:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:01:31.167452 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38761
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:38761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:31.531247 7830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:31.531605 7830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:31.531769 7830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:31.540093 7830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:31.540421 7830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:31.540616 7830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:31.540751 7830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:31.540831 7830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:01:31.552018 7830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:38761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:31.553539 7830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:31.554950 7830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:31.569938 7835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:31.570030 7836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:31.569976 7838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:31.571615 7830 server_base.cc:1034] running on GCE node
I20260927 22:01:31.572634 7830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:31.574589 7830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:31.575963 7830 hybrid_clock.cc:648] HybridClock initialized: now 1790546491575885 us; error 70 us; skew 500 ppm
I20260927 22:01:31.576449 7830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:31 dist-test-slave-sg4j krb5kdc[7817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546491, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:31.587118 7830 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:01:31.588847 7830 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:31.588965 7830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:31.589344 7830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:31.592801 7830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "1f764cf32d7043a7883b6de1b57b4350"
format_stamp: "Formatted at 2026-09-27 22:01:31 on dist-test-slave-sg4j"
server_key: "a506bca04b50f334ff67de88c446bd67"
server_key_iv: "cc9f199eaaf25951907a37270289683f"
server_key_version: "encryptionkey@0"
I20260927 22:01:31.593783 7830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "1f764cf32d7043a7883b6de1b57b4350"
format_stamp: "Formatted at 2026-09-27 22:01:31 on dist-test-slave-sg4j"
server_key: "a506bca04b50f334ff67de88c446bd67"
server_key_iv: "cc9f199eaaf25951907a37270289683f"
server_key_version: "encryptionkey@0"
I20260927 22:01:31.600443 7830 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:01:31.606366 7844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:31.608821 7830 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:01:31.609319 7830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "1f764cf32d7043a7883b6de1b57b4350"
format_stamp: "Formatted at 2026-09-27 22:01:31 on dist-test-slave-sg4j"
server_key: "a506bca04b50f334ff67de88c446bd67"
server_key_iv: "cc9f199eaaf25951907a37270289683f"
server_key_version: "encryptionkey@0"
I20260927 22:01:31.609779 7830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:31.637915 7830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:31.644811 7830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:31.645306 7830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:31.670140 7830 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:38761
I20260927 22:01:31.670318 7895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:38761 every 8 connection(s)
I20260927 22:01:31.671473 7830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:31.674337 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 7830
I20260927 22:01:31.674645 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:31.675514 3708 external_mini_cluster.cc:1428] Setting key 8f2c968a617ad91ed54df4a2ee6c974d
I20260927 22:01:31.677505 7896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:31.693177 7896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350: Bootstrap starting.
Sep 27 22:01:31 dist-test-slave-sg4j krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:31.697113 7896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:31.698660 7896 log.cc:824] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:31.703132 7896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350: No bootstrap required, opened a new log
I20260927 22:01:31.705595 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:31.678444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.3.159.62:38761)
0927 22:01:31.679463 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:31.679483 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:31.679538 (+ 55us) server_negotiation.cc:407] Connection header received
0927 22:01:31.681504 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:31.681538 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:31.682177 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:31.682718 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:31.684728 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:31.686887 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:31.688673 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:31.690376 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:31.690667 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:31.695782 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:31.695794 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:31.695816 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:31.695879 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:31.700119 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:31.701222 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:31.701231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:31.701242 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:31.701514 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:31.702026 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:31.702031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:31.702034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:31.702783 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:31.703089 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:31.703622 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 22:01:31.704139 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":434,"threads_started":1}
I20260927 22:01:31.712721 7896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f764cf32d7043a7883b6de1b57b4350" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38761 } }
I20260927 22:01:31.713114 7896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:31.713178 7896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f764cf32d7043a7883b6de1b57b4350, State: Initialized, Role: FOLLOWER
I20260927 22:01:31.713948 7896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f764cf32d7043a7883b6de1b57b4350" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38761 } }
I20260927 22:01:31.714419 7896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:31.714596 7896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:31.714813 7896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:31.719108 7896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f764cf32d7043a7883b6de1b57b4350" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38761 } }
I20260927 22:01:31.720082 7896 leader_election.cc:302] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f764cf32d7043a7883b6de1b57b4350; no voters:
I20260927 22:01:31.721544 7896 leader_election.cc:288] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:31.721863 7901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:31.723259 7901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [term 1 LEADER]: Becoming Leader. State: Replica: 1f764cf32d7043a7883b6de1b57b4350, State: Running, Role: LEADER
I20260927 22:01:31.724254 7901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f764cf32d7043a7883b6de1b57b4350" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38761 } }
I20260927 22:01:31.725538 7896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:31.729053 7902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f764cf32d7043a7883b6de1b57b4350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f764cf32d7043a7883b6de1b57b4350" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38761 } } }
I20260927 22:01:31.729596 7902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:31.728406 7903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f764cf32d7043a7883b6de1b57b4350. Latest consensus state: current_term: 1 leader_uuid: "1f764cf32d7043a7883b6de1b57b4350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f764cf32d7043a7883b6de1b57b4350" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38761 } } }
I20260927 22:01:31.730715 7903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:31.732296 7906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:31.741951 7906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:31.757097 7906 catalog_manager.cc:1347] Generated new cluster ID: 981682786aab4ba08ceee0f081192da5
I20260927 22:01:31.757335 7906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:31.768281 7906 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:31.770295 7906 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:31.785359 7906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350: Generated new TSK 0
I20260927 22:01:31.786818 7906 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:01:31.863586 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38761
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:32.331331 7922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:32.331727 7922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:32.331844 7922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:32.340663 7922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:32.340816 7922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:32.340962 7922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:01:32.352845 7922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:32.354686 7922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:32.356453 7922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:32.373816 7930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:32.373899 7928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:32.374140 7922 server_base.cc:1034] running on GCE node
W20260927 22:01:32.373950 7927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:32.375659 7922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:32.376943 7922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:32.378232 7922 hybrid_clock.cc:648] HybridClock initialized: now 1790546492378119 us; error 96 us; skew 500 ppm
I20260927 22:01:32.378733 7922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:32 dist-test-slave-sg4j krb5kdc[7817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546492, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:32.383328 7922 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:01:32.385080 7922 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:32.385251 7922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:32.385748 7922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:32.389708 7922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "53e6c92262a64273a607189aec530009"
format_stamp: "Formatted at 2026-09-27 22:01:32 on dist-test-slave-sg4j"
server_key: "fac40c0c49bb176fa163094b7eec1c2b"
server_key_iv: "30f86e438cc8b9dcda145065e872d16e"
server_key_version: "encryptionkey@0"
I20260927 22:01:32.390839 7922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "53e6c92262a64273a607189aec530009"
format_stamp: "Formatted at 2026-09-27 22:01:32 on dist-test-slave-sg4j"
server_key: "fac40c0c49bb176fa163094b7eec1c2b"
server_key_iv: "30f86e438cc8b9dcda145065e872d16e"
server_key_version: "encryptionkey@0"
I20260927 22:01:32.398224 7922 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 22:01:32.403043 7936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:32.405156 7922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:01:32.405455 7922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "53e6c92262a64273a607189aec530009"
format_stamp: "Formatted at 2026-09-27 22:01:32 on dist-test-slave-sg4j"
server_key: "fac40c0c49bb176fa163094b7eec1c2b"
server_key_iv: "30f86e438cc8b9dcda145065e872d16e"
server_key_version: "encryptionkey@0"
I20260927 22:01:32.405745 7922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:32.434813 7922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:32.440385 7922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:32.440725 7922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:32.442098 7922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:32.443965 7922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:32.444077 7922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:32.444208 7922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:32.444301 7922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:32.487591 7922 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:35709
I20260927 22:01:32.487721 8048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:35709 every 8 connection(s)
I20260927 22:01:32.488791 7922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:01:32.494153 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 7922
I20260927 22:01:32.494388 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:01:32.495157 3708 external_mini_cluster.cc:1428] Setting key d0ee262663913d458b49236154c63601
Sep 27 22:01:32 dist-test-slave-sg4j krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546492, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:32.515496 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:32.492121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:59419 (local address 127.3.159.62:38761)
0927 22:01:32.492688 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:32.492694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:32.494171 (+ 1477us) server_negotiation.cc:407] Connection header received
0927 22:01:32.496668 (+ 2497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:32.496673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:32.496769 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:32.496907 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:32.498819 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:32.499574 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:32.501257 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:32.501723 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:32.501870 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:32.506393 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:32.506404 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:32.506414 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:32.506472 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:32.510690 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:32.512033 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:32.512040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:32.512046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:32.512261 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:32.512912 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:32.512921 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:32.512928 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:32.513435 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:32.513628 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:32.514676 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0927 22:01:32.514979 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":309,"threads_started":1}
I20260927 22:01:32.516769 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:32.492887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38761 (local address 127.3.159.1:59419)
0927 22:01:32.493954 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:32.494011 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:32.496295 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:32.497226 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:32.497240 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:32.497934 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:32.498619 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:32.498640 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:32.499759 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:32.499764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:32.501077 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:32.501090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:32.501978 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:32.501988 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:32.502234 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:32.502986 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:32.503010 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:32.506040 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:32.510953 (+ 4913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:32.510966 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:32.510989 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:32.511812 (+ 823us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:32.512454 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:32.512460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:32.512463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:32.512728 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:32.513610 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:32.513618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:32.514087 (+ 469us) client_negotiation.cc:769] Sending connection context
0927 22:01:32.514451 (+ 364us) client_negotiation.cc:240] Negotiation successful
0927 22:01:32.515155 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":478,"threads_started":1}
I20260927 22:01:32.518924 8049 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38761
I20260927 22:01:32.519404 8049 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:01:32.520370 8049 heartbeater.cc:511] Master 127.3.159.62:38761 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:01:32.523553 7861 ts_manager.cc:194] Registered new tserver with Master: 53e6c92262a64273a607189aec530009 (127.3.159.1:35709)
I20260927 22:01:32.526206 7861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:59419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:01:32.553932 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38761
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:32.979910 8055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:32.980345 8055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:32.980535 8055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:32.988991 8055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:32.989184 8055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:32.989413 8055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:01:33.000857 8055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:33.002825 8055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:33.004621 8055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:33.021842 8060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:33.021978 8061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:33.021978 8063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:33.024480 8055 server_base.cc:1034] running on GCE node
I20260927 22:01:33.025597 8055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:33.027205 8055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:33.030799 8055 hybrid_clock.cc:648] HybridClock initialized: now 1790546493030668 us; error 110 us; skew 500 ppm
I20260927 22:01:33.031353 8055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:33 dist-test-slave-sg4j krb5kdc[7817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:33.036139 8055 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:01:33.038046 8055 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:33.038220 8055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:33.038638 8055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:33.042721 8055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "89a7b359d0d245ffa43aa41d593cd6e6"
format_stamp: "Formatted at 2026-09-27 22:01:33 on dist-test-slave-sg4j"
server_key: "1d91aa27a18f7e6733ab01a7a7a1d034"
server_key_iv: "31f903ce07b6d70046b80e14972e998e"
server_key_version: "encryptionkey@0"
I20260927 22:01:33.043977 8055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "89a7b359d0d245ffa43aa41d593cd6e6"
format_stamp: "Formatted at 2026-09-27 22:01:33 on dist-test-slave-sg4j"
server_key: "1d91aa27a18f7e6733ab01a7a7a1d034"
server_key_iv: "31f903ce07b6d70046b80e14972e998e"
server_key_version: "encryptionkey@0"
I20260927 22:01:33.051448 8055 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260927 22:01:33.056814 8069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:33.059084 8055 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 22:01:33.059396 8055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "89a7b359d0d245ffa43aa41d593cd6e6"
format_stamp: "Formatted at 2026-09-27 22:01:33 on dist-test-slave-sg4j"
server_key: "1d91aa27a18f7e6733ab01a7a7a1d034"
server_key_iv: "31f903ce07b6d70046b80e14972e998e"
server_key_version: "encryptionkey@0"
I20260927 22:01:33.059803 8055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:33.088856 8055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:33.096297 8055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:33.096714 8055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:33.098127 8055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:33.099968 8055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:33.100114 8055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:33.100281 8055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:33.100384 8055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:33.144969 8055 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:38009
I20260927 22:01:33.145083 8181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:38009 every 8 connection(s)
I20260927 22:01:33.146153 8055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:01:33.152719 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 8055
I20260927 22:01:33.152959 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:01:33.153967 3708 external_mini_cluster.cc:1428] Setting key 37bb800d8ba5544d19812b8d8d8bfa1e
Sep 27 22:01:33 dist-test-slave-sg4j krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:33.173329 8184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:33.149220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:41275 (local address 127.3.159.62:38761)
0927 22:01:33.149878 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:33.149884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:33.150983 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 22:01:33.153825 (+ 2842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:33.153832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:33.153966 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:33.154107 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:33.156732 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.157882 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.159960 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.160756 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.160965 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:33.166603 (+ 5638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:33.166609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:33.166612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:33.166652 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:33.169879 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:33.170832 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:33.170838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:33.170842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:33.170981 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:33.171437 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:33.171445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:33.171448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:33.171744 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:33.171904 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:33.172711 (+ 807us) server_negotiation.cc:299] Negotiation successful
0927 22:01:33.172957 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":378,"threads_started":1}
I20260927 22:01:33.175024 8185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:33.149820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38761 (local address 127.3.159.2:41275)
0927 22:01:33.150701 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:33.150808 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:33.153324 (+ 2516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:33.154448 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:33.154461 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:33.155381 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:33.156450 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:33.156489 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.158093 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.158097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:33.159735 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:33.159752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.161089 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.161094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:33.161349 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:33.162415 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:33.162446 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:33.166264 (+ 3818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:33.170098 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:33.170120 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:33.170136 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:33.170654 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:33.171107 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:33.171112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:33.171116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:33.171318 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:33.171877 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:33.171887 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:33.172324 (+ 437us) client_negotiation.cc:769] Sending connection context
0927 22:01:33.172634 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 22:01:33.173233 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":337,"threads_started":1}
I20260927 22:01:33.177116 8182 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38761
I20260927 22:01:33.177538 8182 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:33.178332 8182 heartbeater.cc:511] Master 127.3.159.62:38761 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
I20260927 22:01:33.180259 7861 ts_manager.cc:194] Registered new tserver with Master: 89a7b359d0d245ffa43aa41d593cd6e6 (127.3.159.2:38009)
I20260927 22:01:33.181403 7861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:41275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:01:33.209854 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38761
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:33109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:33.530335 8049 heartbeater.cc:503] Master 127.3.159.62:38761 was elected leader, sending a full tablet report...
W20260927 22:01:33.630708 8188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:33.631148 8188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:33.631359 8188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:33.639870 8188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:33.640100 8188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:33.640334 8188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:01:33.652961 8188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:33.655006 8188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:33.656788 8188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:33.673610 8194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:33.673578 8193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:33.674387 8196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:33.679652 8188 server_base.cc:1034] running on GCE node
I20260927 22:01:33.680600 8188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:33.682133 8188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:33.685539 8188 hybrid_clock.cc:648] HybridClock initialized: now 1790546493685444 us; error 85 us; skew 500 ppm
I20260927 22:01:33.686048 8188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:33 dist-test-slave-sg4j krb5kdc[7817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:33.690784 8188 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:01:33.692553 8188 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:33.692718 8188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:33.693114 8188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:33.697037 8188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "8f046a2d494243779934900625af8f62"
format_stamp: "Formatted at 2026-09-27 22:01:33 on dist-test-slave-sg4j"
server_key: "41773255e4633252aa79a546f209a739"
server_key_iv: "0b763ace207c5281122aed3d4606477d"
server_key_version: "encryptionkey@0"
I20260927 22:01:33.698180 8188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "8f046a2d494243779934900625af8f62"
format_stamp: "Formatted at 2026-09-27 22:01:33 on dist-test-slave-sg4j"
server_key: "41773255e4633252aa79a546f209a739"
server_key_iv: "0b763ace207c5281122aed3d4606477d"
server_key_version: "encryptionkey@0"
I20260927 22:01:33.704842 8188 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 22:01:33.710206 8202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:33.712778 8188 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260927 22:01:33.713074 8188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "8f046a2d494243779934900625af8f62"
format_stamp: "Formatted at 2026-09-27 22:01:33 on dist-test-slave-sg4j"
server_key: "41773255e4633252aa79a546f209a739"
server_key_iv: "0b763ace207c5281122aed3d4606477d"
server_key_version: "encryptionkey@0"
I20260927 22:01:33.713384 8188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:33.729816 8188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:33.735844 8188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:33.736202 8188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:33.737557 8188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:33.739507 8188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:33.739681 8188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:33.739863 8188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:33.739964 8188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:33.783401 8188 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:33139
I20260927 22:01:33.783524 8314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:33139 every 8 connection(s)
I20260927 22:01:33.784598 8188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:01:33.789793 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 8188
I20260927 22:01:33.790014 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:01:33.790591 3708 external_mini_cluster.cc:1428] Setting key 6b5d187fce49187880538f6cd8238d13
Sep 27 22:01:33 dist-test-slave-sg4j krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546493, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:33.811959 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:33.788116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:55105 (local address 127.3.159.62:38761)
0927 22:01:33.788773 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:33.788778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:33.790581 (+ 1803us) server_negotiation.cc:407] Connection header received
0927 22:01:33.793588 (+ 3007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:33.793594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:33.793697 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:33.793834 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:33.796231 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.797199 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.799153 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.799669 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.799805 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:33.804072 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:33.804079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:33.804082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:33.804133 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:33.807939 (+ 3806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:33.809028 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:33.809033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:33.809035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:33.809189 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:33.809859 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:33.809863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:33.809865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:33.810355 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:33.810497 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:33.811435 (+ 938us) server_negotiation.cc:299] Negotiation successful
0927 22:01:33.811636 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":360,"threads_started":1}
I20260927 22:01:33.813057 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:33.788910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38761 (local address 127.3.159.3:55105)
0927 22:01:33.790337 (+ 1427us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:33.790397 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:33.792943 (+ 2546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:33.794327 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:33.794344 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:33.795170 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:33.795909 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:33.795934 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.797519 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.797524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:33.798941 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:33.798957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.799955 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.799962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:33.800199 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:33.801069 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:33.801094 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:33.803745 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:33.808171 (+ 4426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:33.808182 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:33.808203 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:33.808818 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:33.809317 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:33.809320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:33.809322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:33.809639 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:33.810565 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:33.810571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:33.810976 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 22:01:33.811231 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 22:01:33.811723 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":694,"threads_started":1}
I20260927 22:01:33.814955 8315 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38761
I20260927 22:01:33.815335 8315 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:33.816272 8315 heartbeater.cc:511] Master 127.3.159.62:38761 requested a full tablet report, sending...
I20260927 22:01:33.818158 7861 ts_manager.cc:194] Registered new tserver with Master: 8f046a2d494243779934900625af8f62 (127.3.159.3:33139)
I20260927 22:01:33.819283 7861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:55105
I20260927 22:01:33.821277 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:01:33.843899 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:33.827704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.3.159.62:38761)
0927 22:01:33.828017 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:33.828025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:33.828803 (+ 778us) server_negotiation.cc:407] Connection header received
0927 22:01:33.829037 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:33.829043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:33.829129 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:33.829249 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:33.830951 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.831817 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.833141 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.833766 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.833879 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:33.837499 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:33.837505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:33.837509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:33.837557 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:33.840592 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:33.841580 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:33.841586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:33.841590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:33.841744 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:33.842163 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:33.842171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:33.842176 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:33.842584 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:33.842857 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:33.843234 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 22:01:33.843524 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 22:01:33.849467 7861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:01:33.858000 7861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:33.888063 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:33.877862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.3.159.2:38009)
0927 22:01:33.878912 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:33.878922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:33.878954 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:01:33.879061 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:33.879071 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:33.879377 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:33.880476 (+ 1099us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:33.880713 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.882202 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.885454 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.886498 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.886692 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:33.886928 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:33.887351 (+ 423us) server_negotiation.cc:299] Negotiation successful
0927 22:01:33.887635 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":1944,"threads_started":1}
I20260927 22:01:33.888058 8328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:33.875901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35709 (local address 127.0.0.1:49824)
0927 22:01:33.876987 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:33.877028 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:33.877234 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:33.878672 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:33.878680 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:33.878716 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:33.879285 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:33.879301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.881614 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.881621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:33.884484 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:33.884497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.887113 (+ 2616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.887119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:33.887321 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:33.887349 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:01:33.887434 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:01:33.887570 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":673,"threads_started":1}
I20260927 22:01:33.888096 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:33.877514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:38009 (local address 127.0.0.1:47148)
0927 22:01:33.878193 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:33.878216 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:33.878358 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:33.879621 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:33.879626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:33.879656 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:33.880158 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:33.880172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.882409 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.882417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:33.885197 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:33.885214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.886743 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.886750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:33.886992 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:33.887079 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 22:01:33.887247 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 22:01:33.887496 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":30,"thread_start_us":343,"threads_started":1}
I20260927 22:01:33.889113 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:33.876269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.3.159.1:35709)
0927 22:01:33.877144 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:33.877152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:33.877185 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:01:33.877553 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:33.877563 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:33.878015 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:33.878308 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:33.879530 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.881457 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.884676 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.886551 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.886704 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:33.887014 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:33.888500 (+ 1486us) server_negotiation.cc:299] Negotiation successful
0927 22:01:33.888763 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":458,"threads_started":1}
I20260927 22:01:33.891914 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:33.878487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53034 (local address 127.3.159.3:33139)
0927 22:01:33.881467 (+ 2980us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:33.881474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:33.881504 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:01:33.881628 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:33.881636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:33.881937 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:33.882173 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:33.883283 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.884968 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.887891 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.890580 (+ 2689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.890748 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:33.890993 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:33.891365 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 22:01:33.891558 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2753,"thread_start_us":2598,"threads_started":1}
I20260927 22:01:33.892238 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:33.878760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:33139 (local address 127.0.0.1:53034)
0927 22:01:33.880593 (+ 1833us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:33.880621 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:33.880787 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:33.882201 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:33.882207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:33.882240 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:33.882909 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:33.882924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.885172 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.885178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:33.887626 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:33.887643 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:33.890867 (+ 3224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:33.890876 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:33.891115 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:33.891159 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:01:33.891308 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:01:33.891511 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1665,"mutex_wait_us":22,"thread_start_us":1514,"threads_started":1}
I20260927 22:01:33.893695 7984 tablet_service.cc:1467] Processing CreateTablet for tablet 739a4ef2eddf4c05a16e996435aeb050 (DEFAULT_TABLE table=test-table [id=3ec401a2d0d946a5b15501c5ff27eb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:33.893913 8117 tablet_service.cc:1467] Processing CreateTablet for tablet 739a4ef2eddf4c05a16e996435aeb050 (DEFAULT_TABLE table=test-table [id=3ec401a2d0d946a5b15501c5ff27eb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:33.895385 7984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739a4ef2eddf4c05a16e996435aeb050. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:33.895694 8117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739a4ef2eddf4c05a16e996435aeb050. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:33.896486 8250 tablet_service.cc:1467] Processing CreateTablet for tablet 739a4ef2eddf4c05a16e996435aeb050 (DEFAULT_TABLE table=test-table [id=3ec401a2d0d946a5b15501c5ff27eb8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:33.897994 8250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 739a4ef2eddf4c05a16e996435aeb050. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:33.911041 8334 tablet_bootstrap.cc:492] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: Bootstrap starting.
I20260927 22:01:33.914268 8335 tablet_bootstrap.cc:492] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: Bootstrap starting.
I20260927 22:01:33.915665 8334 tablet_bootstrap.cc:654] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:33.916590 8336 tablet_bootstrap.cc:492] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: Bootstrap starting.
I20260927 22:01:33.917551 8334 log.cc:824] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:33.918743 8335 tablet_bootstrap.cc:654] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:33.920501 8335 log.cc:824] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:33.920855 8336 tablet_bootstrap.cc:654] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:33.922288 8336 log.cc:824] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:33.924713 8335 tablet_bootstrap.cc:492] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: No bootstrap required, opened a new log
I20260927 22:01:33.924849 8334 tablet_bootstrap.cc:492] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: No bootstrap required, opened a new log
I20260927 22:01:33.925098 8335 ts_tablet_manager.cc:1397] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260927 22:01:33.925230 8334 ts_tablet_manager.cc:1397] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260927 22:01:33.925948 8336 tablet_bootstrap.cc:492] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: No bootstrap required, opened a new log
I20260927 22:01:33.926244 8336 ts_tablet_manager.cc:1397] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260927 22:01:33.934187 8335 raft_consensus.cc:348] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e6c92262a64273a607189aec530009" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35709 } } peers { permanent_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38009 } } peers { permanent_uuid: "8f046a2d494243779934900625af8f62" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33139 } }
I20260927 22:01:33.934952 8335 raft_consensus.cc:374] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:33.935084 8335 raft_consensus.cc:729] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e6c92262a64273a607189aec530009, State: Initialized, Role: FOLLOWER
I20260927 22:01:33.935956 8335 consensus_queue.cc:260] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e6c92262a64273a607189aec530009" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35709 } } peers { permanent_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38009 } } peers { permanent_uuid: "8f046a2d494243779934900625af8f62" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33139 } }
I20260927 22:01:33.937594 8336 raft_consensus.cc:348] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e6c92262a64273a607189aec530009" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35709 } } peers { permanent_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38009 } } peers { permanent_uuid: "8f046a2d494243779934900625af8f62" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33139 } }
I20260927 22:01:33.937594 8334 raft_consensus.cc:348] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e6c92262a64273a607189aec530009" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35709 } } peers { permanent_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38009 } } peers { permanent_uuid: "8f046a2d494243779934900625af8f62" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33139 } }
I20260927 22:01:33.938148 8336 raft_consensus.cc:374] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:33.938149 8334 raft_consensus.cc:374] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:33.938179 8335 ts_tablet_manager.cc:1428] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: Time spent starting tablet: real 0.013s user 0.007s sys 0.008s
I20260927 22:01:33.938246 8334 raft_consensus.cc:729] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89a7b359d0d245ffa43aa41d593cd6e6, State: Initialized, Role: FOLLOWER
I20260927 22:01:33.938246 8336 raft_consensus.cc:729] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f046a2d494243779934900625af8f62, State: Initialized, Role: FOLLOWER
I20260927 22:01:33.939000 8334 consensus_queue.cc:260] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e6c92262a64273a607189aec530009" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35709 } } peers { permanent_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38009 } } peers { permanent_uuid: "8f046a2d494243779934900625af8f62" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33139 } }
I20260927 22:01:33.938995 8336 consensus_queue.cc:260] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e6c92262a64273a607189aec530009" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35709 } } peers { permanent_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38009 } } peers { permanent_uuid: "8f046a2d494243779934900625af8f62" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33139 } }
I20260927 22:01:33.940863 8315 heartbeater.cc:503] Master 127.3.159.62:38761 was elected leader, sending a full tablet report...
I20260927 22:01:33.940984 8182 heartbeater.cc:503] Master 127.3.159.62:38761 was elected leader, sending a full tablet report...
I20260927 22:01:33.941267 8334 ts_tablet_manager.cc:1428] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260927 22:01:33.943322 8336 ts_tablet_manager.cc:1428] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: Time spent starting tablet: real 0.017s user 0.007s sys 0.008s
W20260927 22:01:33.992326 8050 tablet.cc:2369] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:01:34.036798 8316 tablet.cc:2369] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:34.120067 8341 raft_consensus.cc:482] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:34.120493 8341 raft_consensus.cc:504] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e6c92262a64273a607189aec530009" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35709 } } peers { permanent_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38009 } } peers { permanent_uuid: "8f046a2d494243779934900625af8f62" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33139 } }
I20260927 22:01:34.122891 8341 leader_election.cc:288] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e6c92262a64273a607189aec530009 (127.3.159.1:35709), 8f046a2d494243779934900625af8f62 (127.3.159.3:33139)
I20260927 22:01:34.134090 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:34.125296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:55591 (local address 127.3.159.3:33139)
0927 22:01:34.125596 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:34.125606 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:34.125806 (+ 200us) server_negotiation.cc:407] Connection header received
0927 22:01:34.126101 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:34.126110 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:34.126239 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:34.126373 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:34.126951 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.128030 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.130964 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.132431 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.132661 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:34.132798 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:34.133602 (+ 804us) server_negotiation.cc:299] Negotiation successful
0927 22:01:34.133799 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260927 22:01:34.134101 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:34.123748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:35641 (local address 127.3.159.1:35709)
0927 22:01:34.124098 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:34.124103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:34.124772 (+ 669us) server_negotiation.cc:407] Connection header received
0927 22:01:34.125057 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:34.125068 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:34.125198 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:34.125264 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:34.126298 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.127599 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.130960 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.132431 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.132658 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:34.132798 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:34.133652 (+ 854us) server_negotiation.cc:299] Negotiation successful
0927 22:01:34.133843 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260927 22:01:34.134155 8344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:34.124790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:33139 (local address 127.3.159.2:55591)
0927 22:01:34.125636 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:34.125682 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:34.125925 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:34.126433 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:34.126438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:34.126472 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:34.126774 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:34.126781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.128195 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.128205 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:34.130548 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:34.130651 (+ 103us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.132903 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.132911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:34.133218 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:34.133351 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 22:01:34.133518 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 22:01:34.133685 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":243,"thread_start_us":357,"threads_started":1}
I20260927 22:01:34.134976 8343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:34.123715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35709 (local address 127.3.159.2:35641)
0927 22:01:34.124628 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:34.124651 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:34.124811 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:34.125526 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:34.125529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:34.125583 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:34.126082 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:34.126096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.127867 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.127871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:34.130553 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:34.130666 (+ 113us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.132903 (+ 2237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.132911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:34.133241 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:34.133351 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 22:01:34.134515 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0927 22:01:34.134648 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":171,"thread_start_us":538,"threads_started":1}
I20260927 22:01:34.135571 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739a4ef2eddf4c05a16e996435aeb050" candidate_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f046a2d494243779934900625af8f62" is_pre_election: true
I20260927 22:01:34.136227 8270 raft_consensus.cc:2454] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89a7b359d0d245ffa43aa41d593cd6e6 in term 0.
I20260927 22:01:34.136328 8004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739a4ef2eddf4c05a16e996435aeb050" candidate_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e6c92262a64273a607189aec530009" is_pre_election: true
I20260927 22:01:34.136978 8004 raft_consensus.cc:2454] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89a7b359d0d245ffa43aa41d593cd6e6 in term 0.
I20260927 22:01:34.137120 8071 leader_election.cc:302] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89a7b359d0d245ffa43aa41d593cd6e6, 8f046a2d494243779934900625af8f62; no voters:
I20260927 22:01:34.137722 8341 raft_consensus.cc:2781] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:34.137907 8341 raft_consensus.cc:482] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:34.138031 8341 raft_consensus.cc:3037] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:34.140578 8341 raft_consensus.cc:504] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e6c92262a64273a607189aec530009" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35709 } } peers { permanent_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38009 } } peers { permanent_uuid: "8f046a2d494243779934900625af8f62" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33139 } }
I20260927 22:01:34.141367 8341 leader_election.cc:288] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [CANDIDATE]: Term 1 election: Requested vote from peers 53e6c92262a64273a607189aec530009 (127.3.159.1:35709), 8f046a2d494243779934900625af8f62 (127.3.159.3:33139)
I20260927 22:01:34.142222 8270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739a4ef2eddf4c05a16e996435aeb050" candidate_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f046a2d494243779934900625af8f62"
I20260927 22:01:34.142223 8004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "739a4ef2eddf4c05a16e996435aeb050" candidate_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e6c92262a64273a607189aec530009"
I20260927 22:01:34.142514 8270 raft_consensus.cc:3037] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:34.142549 8004 raft_consensus.cc:3037] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:34.146381 8270 raft_consensus.cc:2454] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89a7b359d0d245ffa43aa41d593cd6e6 in term 1.
I20260927 22:01:34.146431 8004 raft_consensus.cc:2454] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89a7b359d0d245ffa43aa41d593cd6e6 in term 1.
I20260927 22:01:34.147058 8071 leader_election.cc:302] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e6c92262a64273a607189aec530009, 89a7b359d0d245ffa43aa41d593cd6e6; no voters:
I20260927 22:01:34.147560 8341 raft_consensus.cc:2781] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:34.148280 8341 raft_consensus.cc:686] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 1 LEADER]: Becoming Leader. State: Replica: 89a7b359d0d245ffa43aa41d593cd6e6, State: Running, Role: LEADER
W20260927 22:01:34.148759 8183 tablet.cc:2369] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:34.150068 8341 consensus_queue.cc:237] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e6c92262a64273a607189aec530009" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35709 } } peers { permanent_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38009 } } peers { permanent_uuid: "8f046a2d494243779934900625af8f62" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33139 } }
I20260927 22:01:34.156606 7861 catalog_manager.cc:5495] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89a7b359d0d245ffa43aa41d593cd6e6 (127.3.159.2). New cstate: current_term: 1 leader_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e6c92262a64273a607189aec530009" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89a7b359d0d245ffa43aa41d593cd6e6" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f046a2d494243779934900625af8f62" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33139 } health_report { overall_health: UNKNOWN } } }
I20260927 22:01:34.195359 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:34.189123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.3.159.2:38009)
0927 22:01:34.189429 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:34.189438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:34.189468 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:01:34.189610 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:34.189614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:34.189704 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:34.189882 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:34.190440 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.191202 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.192878 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.193332 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.193638 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:34.194228 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:34.194811 (+ 583us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:34.194948 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:34.195062 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 22:01:34.195137 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 22:01:34.205858 8270 raft_consensus.cc:1261] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62 [term 1 FOLLOWER]: Refusing update from remote peer 89a7b359d0d245ffa43aa41d593cd6e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:34.205942 8004 raft_consensus.cc:1261] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009 [term 1 FOLLOWER]: Refusing update from remote peer 89a7b359d0d245ffa43aa41d593cd6e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:34.207228 8345 consensus_queue.cc:1035] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f046a2d494243779934900625af8f62" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:34.207701 8341 consensus_queue.cc:1035] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e6c92262a64273a607189aec530009" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:34.245579 7860 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:34.245949 7860 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:34.250591 7860 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f764cf32d7043a7883b6de1b57b4350: Sending DeleteTablet for 3 replicas of tablet 739a4ef2eddf4c05a16e996435aeb050
I20260927 22:01:34.252162 7984 tablet_service.cc:1514] Processing DeleteTablet for tablet 739a4ef2eddf4c05a16e996435aeb050 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:34 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:49824
I20260927 22:01:34.252385 8250 tablet_service.cc:1514] Processing DeleteTablet for tablet 739a4ef2eddf4c05a16e996435aeb050 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:34 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:53034
I20260927 22:01:34.252378 8117 tablet_service.cc:1514] Processing DeleteTablet for tablet 739a4ef2eddf4c05a16e996435aeb050 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:34 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:47148
I20260927 22:01:34.253355 8358 tablet_replica.cc:330] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: stopping tablet replica
I20260927 22:01:34.254142 8358 raft_consensus.cc:2229] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:34.254966 8358 raft_consensus.cc:2258] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:34.256537 8359 tablet_replica.cc:330] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: stopping tablet replica
I20260927 22:01:34.257352 8359 raft_consensus.cc:2229] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:34.257997 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 7922
I20260927 22:01:34.258147 8359 raft_consensus.cc:2258] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:34.264640 8359 ts_tablet_manager.cc:1905] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:34.264998 8360 tablet_replica.cc:330] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: stopping tablet replica
I20260927 22:01:34.265831 8360 raft_consensus.cc:2229] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:01:34.266739 8360 raft_consensus.cc:2258] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:34.269625 8360 ts_tablet_manager.cc:1905] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:34.273943 8358 ts_tablet_manager.cc:1905] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:34.278465 8360 ts_tablet_manager.cc:1918] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:34.278725 8360 log.cc:1196] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/739a4ef2eddf4c05a16e996435aeb050
I20260927 22:01:34.279469 8360 ts_tablet_manager.cc:1939] T 739a4ef2eddf4c05a16e996435aeb050 P 89a7b359d0d245ffa43aa41d593cd6e6: Deleting consensus metadata
Sep 27 22:01:34 dist-test-slave-sg4j krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:01:34.282095 7846 catalog_manager.cc:4841] TS 89a7b359d0d245ffa43aa41d593cd6e6 (127.3.159.2:38009): tablet 739a4ef2eddf4c05a16e996435aeb050 (table test-table [id=3ec401a2d0d946a5b15501c5ff27eb8b]) successfully deleted
I20260927 22:01:34.282434 8358 ts_tablet_manager.cc:1918] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:34.273321 8359 ts_tablet_manager.cc:1918] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:34.282687 8358 log.cc:1196] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/739a4ef2eddf4c05a16e996435aeb050
I20260927 22:01:34.282909 8359 log.cc:1196] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/739a4ef2eddf4c05a16e996435aeb050
I20260927 22:01:34.283420 8358 ts_tablet_manager.cc:1939] T 739a4ef2eddf4c05a16e996435aeb050 P 53e6c92262a64273a607189aec530009: Deleting consensus metadata
I20260927 22:01:34.283476 8359 ts_tablet_manager.cc:1939] T 739a4ef2eddf4c05a16e996435aeb050 P 8f046a2d494243779934900625af8f62: Deleting consensus metadata
I20260927 22:01:34.285706 7846 catalog_manager.cc:4841] TS 53e6c92262a64273a607189aec530009 (127.3.159.1:35709): tablet 739a4ef2eddf4c05a16e996435aeb050 (table test-table [id=3ec401a2d0d946a5b15501c5ff27eb8b]) successfully deleted
I20260927 22:01:34.286095 7846 catalog_manager.cc:4841] TS 8f046a2d494243779934900625af8f62 (127.3.159.3:33139): tablet 739a4ef2eddf4c05a16e996435aeb050 (table test-table [id=3ec401a2d0d946a5b15501c5ff27eb8b]) successfully deleted
I20260927 22:01:34.291790 8329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:34.259235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.3.159.1:35709)
0927 22:01:34.259455 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:34.259461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:34.269416 (+ 9955us) server_negotiation.cc:407] Connection header received
0927 22:01:34.270059 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:34.270067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:34.270174 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:34.270257 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:34.272146 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.273281 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.275390 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.276111 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.276229 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:34.282666 (+ 6437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:34.282692 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:34.282716 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:34.282776 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:34.287501 (+ 4725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:34.288730 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:34.288741 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:34.288745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:34.288940 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:34.289600 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:34.289605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:34.289608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:34.290136 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:34.290504 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:34.290838 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 22:01:34.291127 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:01:34.292779 8044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:34.408052 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 7922
I20260927 22:01:34.430218 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 8055
Sep 27 22:01:34 dist-test-slave-sg4j krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:01:34.453933 8331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:34.431412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.3.159.2:38009)
0927 22:01:34.431673 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:34.431680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:34.431937 (+ 257us) server_negotiation.cc:407] Connection header received
0927 22:01:34.432209 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:34.432214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:34.432307 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:34.432434 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:34.434036 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.435206 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.437085 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.438249 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.438489 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:34.445149 (+ 6660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:34.445174 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:34.445203 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:34.445286 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:34.450011 (+ 4725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:34.451110 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:34.451121 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:34.451124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:34.451336 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:34.451806 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:34.451811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:34.451815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:34.452363 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:34.452832 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:34.453133 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 22:01:34.453469 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:01:34.454977 8177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:34.579026 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 8055
I20260927 22:01:34.598646 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 8188
Sep 27 22:01:34 dist-test-slave-sg4j krb5kdc[7817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:01:34.617158 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:34.599806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53044 (local address 127.3.159.3:33139)
0927 22:01:34.600119 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:34.600124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:34.600152 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:01:34.600219 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:34.600223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:34.600303 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:34.600550 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:34.602169 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.602984 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.604506 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:34.605023 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:34.605150 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:34.610648 (+ 5498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:34.610672 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:34.610697 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:34.610770 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:34.613976 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:34.614960 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:34.614966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:34.614969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:34.615179 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:34.615612 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:34.615615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:34.615617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:34.616056 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:34.616299 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:34.616636 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 22:01:34.616845 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260927 22:01:34.618048 8310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:34.719877 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 8188
I20260927 22:01:34.739569 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 7830
I20260927 22:01:34.741190 7891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:34.860257 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 7830
2026-09-27T22:01:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5085 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T22:01:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:34Z Disabled control of system clock
I20260927 22:01:34.942716 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:41997
--webserver_interface=127.3.159.62
--webserver_port=0
--builtin_ntp_servers=127.3.159.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:41997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:01:35.338289 8369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:35.338634 8369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:35.338802 8369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:35.347447 8369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:35.347765 8369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:35.347913 8369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:35.348133 8369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:35.348220 8369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:01:35.358711 8369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:41997
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:41997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:35.360198 8369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:35.362848 8369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:35.378973 8375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:35.378895 8374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:35.378952 8377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:35.380172 8369 server_base.cc:1034] running on GCE node
I20260927 22:01:35.381122 8369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:35.383015 8369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:35.384323 8369 hybrid_clock.cc:648] HybridClock initialized: now 1790546495384227 us; error 97 us; skew 500 ppm
I20260927 22:01:35.384835 8369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:01:35.388098 8369 webserver.cc:460] Webserver started at http://127.3.159.62:38731/ using document root <none> and password file <none>
I20260927 22:01:35.388932 8369 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:35.389042 8369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:35.389521 8369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:35.392705 8369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "daa5b76b4b2143bb8dcbfb66b885a741"
format_stamp: "Formatted at 2026-09-27 22:01:35 on dist-test-slave-sg4j"
server_key: "40876e5577017b67e32d714f30704a2d"
server_key_iv: "6167265e982ee486ad47b1dd17248801"
server_key_version: "encryptionkey@0"
I20260927 22:01:35.393675 8369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "daa5b76b4b2143bb8dcbfb66b885a741"
format_stamp: "Formatted at 2026-09-27 22:01:35 on dist-test-slave-sg4j"
server_key: "40876e5577017b67e32d714f30704a2d"
server_key_iv: "6167265e982ee486ad47b1dd17248801"
server_key_version: "encryptionkey@0"
I20260927 22:01:35.400039 8369 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:01:35.404748 8383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:35.406912 8369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 22:01:35.407685 8369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "daa5b76b4b2143bb8dcbfb66b885a741"
format_stamp: "Formatted at 2026-09-27 22:01:35 on dist-test-slave-sg4j"
server_key: "40876e5577017b67e32d714f30704a2d"
server_key_iv: "6167265e982ee486ad47b1dd17248801"
server_key_version: "encryptionkey@0"
I20260927 22:01:35.408056 8369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:35.415403 8369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:35.416435 8369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:35.416806 8369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:35.441048 8369 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:41997
I20260927 22:01:35.441174 8434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:41997 every 8 connection(s)
I20260927 22:01:35.442770 8369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:35.447892 8435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:35.449250 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 8369
I20260927 22:01:35.449570 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:35.450224 3708 external_mini_cluster.cc:1428] Setting key 6aad447f5d2b514dc9075b651a5a6007
I20260927 22:01:35.461895 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:35.453896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.3.159.62:41997)
0927 22:01:35.455298 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:35.455314 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:35.455372 (+ 58us) server_negotiation.cc:407] Connection header received
0927 22:01:35.457274 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:35.457305 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:35.457980 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:35.458401 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:35.459883 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:35.459889 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:35.459905 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:35.459926 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:35.460005 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:35.460024 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:35.460255 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:35.460379 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 22:01:35.460729 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":820,"thread_start_us":499,"threads_started":1}
I20260927 22:01:35.463274 8435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741: Bootstrap starting.
I20260927 22:01:35.467404 8435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:35.468770 8435 log.cc:824] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:35.473292 8435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741: No bootstrap required, opened a new log
I20260927 22:01:35.482229 8435 raft_consensus.cc:348] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa5b76b4b2143bb8dcbfb66b885a741" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41997 } }
I20260927 22:01:35.482831 8435 raft_consensus.cc:374] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:35.482963 8435 raft_consensus.cc:729] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa5b76b4b2143bb8dcbfb66b885a741, State: Initialized, Role: FOLLOWER
I20260927 22:01:35.483911 8435 consensus_queue.cc:260] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa5b76b4b2143bb8dcbfb66b885a741" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41997 } }
I20260927 22:01:35.484298 8435 raft_consensus.cc:388] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:35.484477 8435 raft_consensus.cc:482] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:35.484731 8435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:35.487462 8435 raft_consensus.cc:504] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa5b76b4b2143bb8dcbfb66b885a741" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41997 } }
I20260927 22:01:35.488225 8435 leader_election.cc:302] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daa5b76b4b2143bb8dcbfb66b885a741; no voters:
I20260927 22:01:35.489024 8435 leader_election.cc:288] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:35.489346 8440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:35.490458 8440 raft_consensus.cc:686] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [term 1 LEADER]: Becoming Leader. State: Replica: daa5b76b4b2143bb8dcbfb66b885a741, State: Running, Role: LEADER
I20260927 22:01:35.491204 8440 consensus_queue.cc:237] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa5b76b4b2143bb8dcbfb66b885a741" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41997 } }
I20260927 22:01:35.491992 8435 sys_catalog.cc:564] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:35.494745 8441 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daa5b76b4b2143bb8dcbfb66b885a741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa5b76b4b2143bb8dcbfb66b885a741" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41997 } } }
I20260927 22:01:35.495345 8441 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:35.496395 8442 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader daa5b76b4b2143bb8dcbfb66b885a741. Latest consensus state: current_term: 1 leader_uuid: "daa5b76b4b2143bb8dcbfb66b885a741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa5b76b4b2143bb8dcbfb66b885a741" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41997 } } }
I20260927 22:01:35.497157 8442 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:35.502828 8446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:35.510279 8446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:35.523321 8446 catalog_manager.cc:1347] Generated new cluster ID: 2bc716d96167439fbe33ba28bbe7ad73
I20260927 22:01:35.523492 8446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:35.550174 8446 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:35.550850 8446 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:35.563732 8446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741: Generated new TSK 0
I20260927 22:01:35.565097 8446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:01:35.580395 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41997
--builtin_ntp_servers=127.3.159.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 22:01:36.036161 8459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:36.036492 8459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:36.036654 8459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:36.044624 8459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:36.044757 8459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:36.044940 8459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:01:36.054900 8459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:36.056380 8459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:36.058408 8459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:36.074368 8465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:36.074496 8464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:36.074463 8467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:36.076112 8459 server_base.cc:1034] running on GCE node
I20260927 22:01:36.076936 8459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:36.078275 8459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:36.079524 8459 hybrid_clock.cc:648] HybridClock initialized: now 1790546496079471 us; error 45 us; skew 500 ppm
I20260927 22:01:36.079988 8459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:01:36.083110 8459 webserver.cc:460] Webserver started at http://127.3.159.1:36401/ using document root <none> and password file <none>
I20260927 22:01:36.084038 8459 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:36.084139 8459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:36.084553 8459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:36.088135 8459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "50ace9f4f8324937bb32cd18fcdc0a2a"
format_stamp: "Formatted at 2026-09-27 22:01:36 on dist-test-slave-sg4j"
server_key: "7c08a8a398b0f83c632484d809a01e4a"
server_key_iv: "e7d0c7f196b4b521da175a4cc3dc6aa0"
server_key_version: "encryptionkey@0"
I20260927 22:01:36.089233 8459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "50ace9f4f8324937bb32cd18fcdc0a2a"
format_stamp: "Formatted at 2026-09-27 22:01:36 on dist-test-slave-sg4j"
server_key: "7c08a8a398b0f83c632484d809a01e4a"
server_key_iv: "e7d0c7f196b4b521da175a4cc3dc6aa0"
server_key_version: "encryptionkey@0"
I20260927 22:01:36.096202 8459 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:01:36.101210 8473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:36.103356 8459 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:01:36.103648 8459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "50ace9f4f8324937bb32cd18fcdc0a2a"
format_stamp: "Formatted at 2026-09-27 22:01:36 on dist-test-slave-sg4j"
server_key: "7c08a8a398b0f83c632484d809a01e4a"
server_key_iv: "e7d0c7f196b4b521da175a4cc3dc6aa0"
server_key_version: "encryptionkey@0"
I20260927 22:01:36.103964 8459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:36.111653 8459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:36.112874 8459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:36.113272 8459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:36.114777 8459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:36.116755 8459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:36.116861 8459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:36.116959 8459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:36.117017 8459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:36.160073 8459 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:43201
I20260927 22:01:36.160200 8585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:43201 every 8 connection(s)
I20260927 22:01:36.162039 8459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:01:36.169778 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 8459
I20260927 22:01:36.170044 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:01:36.170931 3708 external_mini_cluster.cc:1428] Setting key 56228289b29ad216490eaef2238a3460
I20260927 22:01:36.174043 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:36.165928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:43277 (local address 127.3.159.62:41997)
0927 22:01:36.166620 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:36.166626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:36.168385 (+ 1759us) server_negotiation.cc:407] Connection header received
0927 22:01:36.170543 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:36.170552 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:36.170677 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:36.170895 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:36.172751 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:36.172758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:36.172764 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:36.172787 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:36.172857 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:36.172874 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:36.173029 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:36.173513 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 22:01:36.173594 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":364,"threads_started":1}
I20260927 22:01:36.175072 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41997
--builtin_ntp_servers=127.3.159.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:01:36.175631 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:36.166837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:41997 (local address 127.3.159.1:43277)
0927 22:01:36.168148 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:36.168204 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:36.170121 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:36.171267 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:36.171284 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:36.171436 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:36.172339 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:01:36.172342 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:36.172389 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:01:36.172499 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:36.173153 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:36.173166 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:36.173178 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:01:36.173574 (+ 396us) client_negotiation.cc:240] Negotiation successful
0927 22:01:36.173924 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":401,"threads_started":1}
I20260927 22:01:36.178294 8586 heartbeater.cc:347] Connected to a master server at 127.3.159.62:41997
I20260927 22:01:36.178817 8586 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:36.179682 8586 heartbeater.cc:511] Master 127.3.159.62:41997 requested a full tablet report, sending...
I20260927 22:01:36.182868 8400 ts_manager.cc:194] Registered new tserver with Master: 50ace9f4f8324937bb32cd18fcdc0a2a (127.3.159.1:43201)
W20260927 22:01:36.606704 8590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:36.607343 8590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:36.607550 8590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:36.618129 8590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:36.618294 8590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:36.618516 8590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:01:36.630363 8590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:36.632164 8590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:36.634603 8590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:36.653060 8595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:36.653196 8596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:36.653225 8598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:36.654574 8590 server_base.cc:1034] running on GCE node
I20260927 22:01:36.655529 8590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:36.656952 8590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:36.658263 8590 hybrid_clock.cc:648] HybridClock initialized: now 1790546496658154 us; error 93 us; skew 500 ppm
I20260927 22:01:36.658733 8590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:01:36.667330 8590 webserver.cc:460] Webserver started at http://127.3.159.2:36205/ using document root <none> and password file <none>
I20260927 22:01:36.668540 8590 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:36.668696 8590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:36.669134 8590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:36.673343 8590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "f63d690933b1484091484839bbd1c4fa"
format_stamp: "Formatted at 2026-09-27 22:01:36 on dist-test-slave-sg4j"
server_key: "cf435b28d9cef6b6d001325685a01551"
server_key_iv: "dca92476a893cb19dcd436e6145e7003"
server_key_version: "encryptionkey@0"
I20260927 22:01:36.674515 8590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "f63d690933b1484091484839bbd1c4fa"
format_stamp: "Formatted at 2026-09-27 22:01:36 on dist-test-slave-sg4j"
server_key: "cf435b28d9cef6b6d001325685a01551"
server_key_iv: "dca92476a893cb19dcd436e6145e7003"
server_key_version: "encryptionkey@0"
I20260927 22:01:36.682188 8590 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 22:01:36.687054 8604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:36.689164 8590 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:01:36.689565 8590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "f63d690933b1484091484839bbd1c4fa"
format_stamp: "Formatted at 2026-09-27 22:01:36 on dist-test-slave-sg4j"
server_key: "cf435b28d9cef6b6d001325685a01551"
server_key_iv: "dca92476a893cb19dcd436e6145e7003"
server_key_version: "encryptionkey@0"
I20260927 22:01:36.689877 8590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:36.696950 8590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:36.698437 8590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:36.698802 8590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:36.700102 8590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:36.702080 8590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:36.702225 8590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:36.702392 8590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:36.702492 8590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:36.744647 8590 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:39251
I20260927 22:01:36.744719 8716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:39251 every 8 connection(s)
I20260927 22:01:36.746380 8590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:01:36.756809 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 8590
I20260927 22:01:36.757063 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:01:36.757524 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:36.750247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:46943 (local address 127.3.159.62:41997)
0927 22:01:36.750903 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:36.750909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:36.752757 (+ 1848us) server_negotiation.cc:407] Connection header received
0927 22:01:36.754739 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:36.754745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:36.754838 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:36.754980 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:36.756558 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:36.756567 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:36.756575 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:36.756600 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:36.756657 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:36.756670 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:36.756826 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:36.757063 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 22:01:36.757144 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":347,"threads_started":1}
I20260927 22:01:36.757764 3708 external_mini_cluster.cc:1428] Setting key e5697102f3e4dc9cfa2b187caf8a3f7b
I20260927 22:01:36.758677 8720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:36.751073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:41997 (local address 127.3.159.2:46943)
0927 22:01:36.752522 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:36.752583 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:36.754379 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:36.755269 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:36.755279 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:36.755394 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:36.756194 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:01:36.756197 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:36.756240 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:01:36.756351 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:36.756785 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:36.756798 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:36.756806 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:01:36.757144 (+ 338us) client_negotiation.cc:240] Negotiation successful
0927 22:01:36.757501 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":536,"threads_started":1}
I20260927 22:01:36.761266 8717 heartbeater.cc:347] Connected to a master server at 127.3.159.62:41997
I20260927 22:01:36.762013 8717 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:36.762156 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41997
--builtin_ntp_servers=127.3.159.20:44529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 22:01:36.763000 8717 heartbeater.cc:511] Master 127.3.159.62:41997 requested a full tablet report, sending...
I20260927 22:01:36.765816 8399 ts_manager.cc:194] Registered new tserver with Master: f63d690933b1484091484839bbd1c4fa (127.3.159.2:39251)
I20260927 22:01:37.186452 8586 heartbeater.cc:503] Master 127.3.159.62:41997 was elected leader, sending a full tablet report...
W20260927 22:01:37.193562 8721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:37.194015 8721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:37.194164 8721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:37.203296 8721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:37.203465 8721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:37.203672 8721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:01:37.214671 8721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:37.216473 8721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:37.218986 8721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:37.235806 8727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:37.235958 8726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:37.235987 8729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:37.237696 8721 server_base.cc:1034] running on GCE node
I20260927 22:01:37.238569 8721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:37.239879 8721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:37.241197 8721 hybrid_clock.cc:648] HybridClock initialized: now 1790546497241119 us; error 51 us; skew 500 ppm
I20260927 22:01:37.241755 8721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:01:37.245888 8721 webserver.cc:460] Webserver started at http://127.3.159.3:40753/ using document root <none> and password file <none>
I20260927 22:01:37.246928 8721 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:37.247092 8721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:37.247522 8721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:37.251593 8721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "2319893a9cf740caa9e1a79312961478"
format_stamp: "Formatted at 2026-09-27 22:01:37 on dist-test-slave-sg4j"
server_key: "cd596e21fa0c6a4fe18b721df67e0e8b"
server_key_iv: "cb733bec0e5207ad250183cfb54bf203"
server_key_version: "encryptionkey@0"
I20260927 22:01:37.252730 8721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "2319893a9cf740caa9e1a79312961478"
format_stamp: "Formatted at 2026-09-27 22:01:37 on dist-test-slave-sg4j"
server_key: "cd596e21fa0c6a4fe18b721df67e0e8b"
server_key_iv: "cb733bec0e5207ad250183cfb54bf203"
server_key_version: "encryptionkey@0"
I20260927 22:01:37.260442 8721 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 22:01:37.265873 8735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:37.268278 8721 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:01:37.268553 8721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "2319893a9cf740caa9e1a79312961478"
format_stamp: "Formatted at 2026-09-27 22:01:37 on dist-test-slave-sg4j"
server_key: "cd596e21fa0c6a4fe18b721df67e0e8b"
server_key_iv: "cb733bec0e5207ad250183cfb54bf203"
server_key_version: "encryptionkey@0"
I20260927 22:01:37.268888 8721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:37.276572 8721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:37.277894 8721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:37.278231 8721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:37.279500 8721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:37.281498 8721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:37.281632 8721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:37.281800 8721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:37.281896 8721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:37.325102 8721 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:43347
I20260927 22:01:37.325207 8847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:43347 every 8 connection(s)
I20260927 22:01:37.326781 8721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:01:37.332528 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 8721
I20260927 22:01:37.332752 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:01:37.333348 3708 external_mini_cluster.cc:1428] Setting key e773440bd0264065cba15837dc5424a1
I20260927 22:01:37.338143 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.330313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:58039 (local address 127.3.159.62:41997)
0927 22:01:37.330940 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:37.330944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:37.332579 (+ 1635us) server_negotiation.cc:407] Connection header received
0927 22:01:37.335218 (+ 2639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:37.335222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:37.335339 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:37.335453 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:37.337254 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:37.337259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:37.337268 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:37.337286 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:37.337374 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:37.337410 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:37.337561 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:37.337754 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 22:01:37.337819 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":400,"threads_started":1}
I20260927 22:01:37.339344 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.330931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:41997 (local address 127.3.159.3:58039)
0927 22:01:37.332322 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:37.332394 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:37.334713 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:37.335828 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:37.335843 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:37.336000 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:37.336810 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:01:37.336813 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:37.336858 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:01:37.337018 (+ 160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:37.337531 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:37.337539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:37.337546 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:01:37.337821 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 22:01:37.338080 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":363,"threads_started":1}
I20260927 22:01:37.341059 8848 heartbeater.cc:347] Connected to a master server at 127.3.159.62:41997
I20260927 22:01:37.341495 8848 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:37.342201 8848 heartbeater.cc:511] Master 127.3.159.62:41997 requested a full tablet report, sending...
I20260927 22:01:37.343890 8399 ts_manager.cc:194] Registered new tserver with Master: 2319893a9cf740caa9e1a79312961478 (127.3.159.3:43347)
I20260927 22:01:37.351316 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:01:37.361123 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.357836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.3.159.62:41997)
0927 22:01:37.358202 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:37.358213 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:37.358696 (+ 483us) server_negotiation.cc:407] Connection header received
0927 22:01:37.358884 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:37.358888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:37.358987 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:37.359104 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:37.360322 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:37.360326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:37.360330 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:37.360356 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:37.360401 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:37.360414 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:37.360553 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:37.360720 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 22:01:37.360779 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260927 22:01:37.367306 8399 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:01:37.375629 8399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:37.412256 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.392448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:43201 (local address 127.0.0.1:35560)
0927 22:01:37.403990 (+ 11542us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:37.404045 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:37.404275 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:37.409191 (+ 4916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:37.409200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:37.409405 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:37.410565 (+ 1160us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:01:37.410570 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:37.410615 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:01:37.410713 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:37.411330 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:37.411345 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:37.411353 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:01:37.411553 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 22:01:37.411660 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11177,"thread_start_us":14493,"threads_started":1}
I20260927 22:01:37.413286 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.392810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:43347 (local address 127.0.0.1:44958)
0927 22:01:37.407284 (+ 14474us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:37.407314 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:37.407522 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:37.410978 (+ 3456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:37.410994 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:37.411025 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:37.411842 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:01:37.411845 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:37.411865 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:01:37.411918 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:37.412533 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:37.412537 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:37.412541 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 22:01:37.412724 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 22:01:37.412766 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14295,"thread_start_us":10562,"threads_started":1}
I20260927 22:01:37.413323 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.392838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.3.159.1:43201)
0927 22:01:37.408365 (+ 15527us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:37.408375 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:37.408404 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:01:37.408481 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:37.408491 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:37.408698 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:37.408856 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:37.410911 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:37.410925 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:37.410944 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:37.410977 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:37.411048 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:37.411073 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:37.411274 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:37.412769 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0927 22:01:37.412916 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15256,"thread_start_us":15103,"threads_started":1}
I20260927 22:01:37.413719 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.393608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.3.159.2:39251)
0927 22:01:37.406371 (+ 12763us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:37.406388 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:37.407933 (+ 1545us) server_negotiation.cc:407] Connection header received
0927 22:01:37.408036 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:37.408046 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:37.408378 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:37.408556 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:37.411289 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:37.411301 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:37.411314 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:37.411347 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:37.411409 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:37.411433 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:37.412248 (+ 815us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:37.413227 (+ 979us) server_negotiation.cc:299] Negotiation successful
0927 22:01:37.413344 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12449,"thread_start_us":12788,"threads_started":1}
I20260927 22:01:37.414235 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.393639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44958 (local address 127.3.159.3:43347)
0927 22:01:37.410465 (+ 16826us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:37.410475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:37.410522 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:01:37.410607 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:37.410616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:37.410790 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:37.410970 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:37.412190 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:37.412199 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:37.412213 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:37.412230 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:37.412283 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:37.412305 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:37.412472 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:37.413721 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0927 22:01:37.413836 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16142,"thread_start_us":15930,"threads_started":1}
I20260927 22:01:37.413513 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.394182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:39251 (local address 127.0.0.1:34210)
0927 22:01:37.407669 (+ 13487us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:37.407691 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:37.407798 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:37.409142 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:37.409151 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:37.410015 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:37.410995 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:01:37.410999 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:37.411014 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:01:37.411051 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:37.412319 (+ 1268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:37.412327 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:37.412336 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:01:37.412488 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 22:01:37.412561 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13337,"thread_start_us":13266,"threads_started":1}
I20260927 22:01:37.418452 8783 tablet_service.cc:1467] Processing CreateTablet for tablet b52d5f888ee74a6eaf44f584d95c5f11 (DEFAULT_TABLE table=test-table [id=2ed7f4e2c3f84f20b0ef485c4e391e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:37.418651 8652 tablet_service.cc:1467] Processing CreateTablet for tablet b52d5f888ee74a6eaf44f584d95c5f11 (DEFAULT_TABLE table=test-table [id=2ed7f4e2c3f84f20b0ef485c4e391e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:37.418651 8521 tablet_service.cc:1467] Processing CreateTablet for tablet b52d5f888ee74a6eaf44f584d95c5f11 (DEFAULT_TABLE table=test-table [id=2ed7f4e2c3f84f20b0ef485c4e391e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:37.419958 8783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52d5f888ee74a6eaf44f584d95c5f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:37.420277 8521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52d5f888ee74a6eaf44f584d95c5f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:37.420286 8652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b52d5f888ee74a6eaf44f584d95c5f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:37.434935 8867 tablet_bootstrap.cc:492] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: Bootstrap starting.
I20260927 22:01:37.439690 8869 tablet_bootstrap.cc:492] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: Bootstrap starting.
I20260927 22:01:37.440371 8867 tablet_bootstrap.cc:654] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:37.441521 8868 tablet_bootstrap.cc:492] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a: Bootstrap starting.
I20260927 22:01:37.442330 8867 log.cc:824] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:37.445669 8869 tablet_bootstrap.cc:654] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:37.447530 8869 log.cc:824] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:37.451115 8868 tablet_bootstrap.cc:654] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:37.453166 8868 log.cc:824] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:37.464318 8867 tablet_bootstrap.cc:492] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: No bootstrap required, opened a new log
I20260927 22:01:37.464610 8869 tablet_bootstrap.cc:492] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: No bootstrap required, opened a new log
I20260927 22:01:37.464802 8867 ts_tablet_manager.cc:1397] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.013s
I20260927 22:01:37.465147 8869 ts_tablet_manager.cc:1397] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.022s
I20260927 22:01:37.466984 8868 tablet_bootstrap.cc:492] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a: No bootstrap required, opened a new log
I20260927 22:01:37.467383 8868 ts_tablet_manager.cc:1397] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.000s
I20260927 22:01:37.476872 8869 raft_consensus.cc:348] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43201 } } peers { permanent_uuid: "2319893a9cf740caa9e1a79312961478" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 43347 } } peers { permanent_uuid: "f63d690933b1484091484839bbd1c4fa" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 39251 } }
I20260927 22:01:37.477500 8869 raft_consensus.cc:374] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:37.477651 8869 raft_consensus.cc:729] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63d690933b1484091484839bbd1c4fa, State: Initialized, Role: FOLLOWER
I20260927 22:01:37.477865 8867 raft_consensus.cc:348] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43201 } } peers { permanent_uuid: "2319893a9cf740caa9e1a79312961478" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 43347 } } peers { permanent_uuid: "f63d690933b1484091484839bbd1c4fa" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 39251 } }
I20260927 22:01:37.478468 8867 raft_consensus.cc:374] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:37.478586 8867 raft_consensus.cc:729] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2319893a9cf740caa9e1a79312961478, State: Initialized, Role: FOLLOWER
I20260927 22:01:37.478658 8869 consensus_queue.cc:260] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43201 } } peers { permanent_uuid: "2319893a9cf740caa9e1a79312961478" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 43347 } } peers { permanent_uuid: "f63d690933b1484091484839bbd1c4fa" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 39251 } }
I20260927 22:01:37.479465 8867 consensus_queue.cc:260] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43201 } } peers { permanent_uuid: "2319893a9cf740caa9e1a79312961478" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 43347 } } peers { permanent_uuid: "f63d690933b1484091484839bbd1c4fa" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 39251 } }
I20260927 22:01:37.480403 8717 heartbeater.cc:503] Master 127.3.159.62:41997 was elected leader, sending a full tablet report...
I20260927 22:01:37.481357 8869 ts_tablet_manager.cc:1428] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260927 22:01:37.481663 8868 raft_consensus.cc:348] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43201 } } peers { permanent_uuid: "2319893a9cf740caa9e1a79312961478" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 43347 } } peers { permanent_uuid: "f63d690933b1484091484839bbd1c4fa" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 39251 } }
I20260927 22:01:37.482571 8868 raft_consensus.cc:374] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:37.482729 8868 raft_consensus.cc:729] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ace9f4f8324937bb32cd18fcdc0a2a, State: Initialized, Role: FOLLOWER
I20260927 22:01:37.483211 8848 heartbeater.cc:503] Master 127.3.159.62:41997 was elected leader, sending a full tablet report...
I20260927 22:01:37.483593 8868 consensus_queue.cc:260] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43201 } } peers { permanent_uuid: "2319893a9cf740caa9e1a79312961478" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 43347 } } peers { permanent_uuid: "f63d690933b1484091484839bbd1c4fa" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 39251 } }
I20260927 22:01:37.485826 8868 ts_tablet_manager.cc:1428] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260927 22:01:37.487373 8867 ts_tablet_manager.cc:1428] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: Time spent starting tablet: real 0.022s user 0.011s sys 0.006s
W20260927 22:01:37.499351 8718 tablet.cc:2369] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:37.570387 8873 raft_consensus.cc:482] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:37.570812 8873 raft_consensus.cc:504] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43201 } } peers { permanent_uuid: "2319893a9cf740caa9e1a79312961478" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 43347 } } peers { permanent_uuid: "f63d690933b1484091484839bbd1c4fa" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 39251 } }
I20260927 22:01:37.572942 8873 leader_election.cc:288] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50ace9f4f8324937bb32cd18fcdc0a2a (127.3.159.1:43201), 2319893a9cf740caa9e1a79312961478 (127.3.159.3:43347)
I20260927 22:01:37.577033 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.573651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:53113 (local address 127.3.159.3:43347)
0927 22:01:37.573887 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:37.573894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:37.574436 (+ 542us) server_negotiation.cc:407] Connection header received
0927 22:01:37.574736 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:37.574741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:37.574799 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:37.574913 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:37.576117 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:37.576126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:37.576133 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:37.576156 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:37.576205 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:37.576223 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:37.576382 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:37.576592 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 22:01:37.576700 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:01:37.577313 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.573848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:49081 (local address 127.3.159.1:43201)
0927 22:01:37.574119 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:37.574129 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:37.574946 (+ 817us) server_negotiation.cc:407] Connection header received
0927 22:01:37.575169 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:37.575174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:37.575257 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:37.575375 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:37.576447 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:37.576455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:37.576461 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:37.576487 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:37.576541 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:37.576561 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:37.576706 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:37.576928 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 22:01:37.577013 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 22:01:37.577282 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.573351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:43347 (local address 127.3.159.2:53113)
0927 22:01:37.574306 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:37.574325 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:37.574562 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:37.574943 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:37.574950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:37.574980 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:37.575813 (+ 833us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:01:37.575818 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:37.575840 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:01:37.575931 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:37.576375 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:37.576382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:37.576388 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 22:01:37.576632 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 22:01:37.576701 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":1026,"threads_started":1}
I20260927 22:01:37.577592 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.573474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:43201 (local address 127.3.159.2:49081)
0927 22:01:37.574797 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:37.574819 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:37.575009 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:37.575396 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:37.575400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:37.575428 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:37.576164 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:01:37.576168 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:37.576185 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:01:37.576226 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:37.576722 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:37.576771 (+ 49us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:37.576776 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 22:01:37.576946 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 22:01:37.577001 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"thread_start_us":449,"threads_started":1}
I20260927 22:01:37.578636 8541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52d5f888ee74a6eaf44f584d95c5f11" candidate_uuid: "f63d690933b1484091484839bbd1c4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" is_pre_election: true
I20260927 22:01:37.578514 8803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52d5f888ee74a6eaf44f584d95c5f11" candidate_uuid: "f63d690933b1484091484839bbd1c4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2319893a9cf740caa9e1a79312961478" is_pre_election: true
W20260927 22:01:37.578959 8849 tablet.cc:2369] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:37.579265 8541 raft_consensus.cc:2454] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f63d690933b1484091484839bbd1c4fa in term 0.
I20260927 22:01:37.579324 8803 raft_consensus.cc:2454] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f63d690933b1484091484839bbd1c4fa in term 0.
I20260927 22:01:37.580135 8608 leader_election.cc:302] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50ace9f4f8324937bb32cd18fcdc0a2a, f63d690933b1484091484839bbd1c4fa; no voters:
I20260927 22:01:37.580585 8873 raft_consensus.cc:2781] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:37.580757 8873 raft_consensus.cc:482] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:37.580869 8873 raft_consensus.cc:3037] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:37.583475 8873 raft_consensus.cc:504] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43201 } } peers { permanent_uuid: "2319893a9cf740caa9e1a79312961478" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 43347 } } peers { permanent_uuid: "f63d690933b1484091484839bbd1c4fa" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 39251 } }
I20260927 22:01:37.584268 8873 leader_election.cc:288] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [CANDIDATE]: Term 1 election: Requested vote from peers 50ace9f4f8324937bb32cd18fcdc0a2a (127.3.159.1:43201), 2319893a9cf740caa9e1a79312961478 (127.3.159.3:43347)
I20260927 22:01:37.584985 8541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52d5f888ee74a6eaf44f584d95c5f11" candidate_uuid: "f63d690933b1484091484839bbd1c4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a"
I20260927 22:01:37.585281 8541 raft_consensus.cc:3037] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:37.585335 8803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b52d5f888ee74a6eaf44f584d95c5f11" candidate_uuid: "f63d690933b1484091484839bbd1c4fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2319893a9cf740caa9e1a79312961478"
I20260927 22:01:37.585776 8803 raft_consensus.cc:3037] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:37.588025 8541 raft_consensus.cc:2454] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f63d690933b1484091484839bbd1c4fa in term 1.
I20260927 22:01:37.588393 8803 raft_consensus.cc:2454] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f63d690933b1484091484839bbd1c4fa in term 1.
I20260927 22:01:37.588786 8608 leader_election.cc:302] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50ace9f4f8324937bb32cd18fcdc0a2a, f63d690933b1484091484839bbd1c4fa; no voters:
I20260927 22:01:37.589267 8873 raft_consensus.cc:2781] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:37.589919 8873 raft_consensus.cc:686] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 1 LEADER]: Becoming Leader. State: Replica: f63d690933b1484091484839bbd1c4fa, State: Running, Role: LEADER
I20260927 22:01:37.590454 8873 consensus_queue.cc:237] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43201 } } peers { permanent_uuid: "2319893a9cf740caa9e1a79312961478" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 43347 } } peers { permanent_uuid: "f63d690933b1484091484839bbd1c4fa" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 39251 } }
I20260927 22:01:37.596901 8400 catalog_manager.cc:5495] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa reported cstate change: term changed from 0 to 1, leader changed from <none> to f63d690933b1484091484839bbd1c4fa (127.3.159.2). New cstate: current_term: 1 leader_uuid: "f63d690933b1484091484839bbd1c4fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2319893a9cf740caa9e1a79312961478" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 43347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f63d690933b1484091484839bbd1c4fa" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 39251 } health_report { overall_health: HEALTHY } } }
I20260927 22:01:37.614470 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.611483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.3.159.2:39251)
0927 22:01:37.611720 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:37.611725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:37.611805 (+ 80us) server_negotiation.cc:407] Connection header received
0927 22:01:37.612097 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:37.612103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:37.612205 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:37.612320 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:37.613573 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:37.613578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:37.613582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:37.613599 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:37.613636 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:37.613649 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:37.613798 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:37.614019 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 22:01:37.614115 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 22:01:37.625108 8803 raft_consensus.cc:1261] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478 [term 1 FOLLOWER]: Refusing update from remote peer f63d690933b1484091484839bbd1c4fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:37.625145 8541 raft_consensus.cc:1261] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a [term 1 FOLLOWER]: Refusing update from remote peer f63d690933b1484091484839bbd1c4fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:37.626390 8878 consensus_queue.cc:1035] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "2319893a9cf740caa9e1a79312961478" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 43347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:37.626952 8873 consensus_queue.cc:1035] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ace9f4f8324937bb32cd18fcdc0a2a" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:37.641552 8881 mvcc.cc:205] Tried to move back new op lower bound from 7334078454259630080 to 7334078454134906880. Current Snapshot: MvccSnapshot[applied={T|T < 7334078454259630080}]
I20260927 22:01:37.662959 8400 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:37.663256 8400 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:37.667604 8400 catalog_manager.cc:5782] T 00000000000000000000000000000000 P daa5b76b4b2143bb8dcbfb66b885a741: Sending DeleteTablet for 3 replicas of tablet b52d5f888ee74a6eaf44f584d95c5f11
I20260927 22:01:37.669278 8783 tablet_service.cc:1514] Processing DeleteTablet for tablet b52d5f888ee74a6eaf44f584d95c5f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:37 UTC) from {username='slave'} at 127.0.0.1:44958
I20260927 22:01:37.669261 8521 tablet_service.cc:1514] Processing DeleteTablet for tablet b52d5f888ee74a6eaf44f584d95c5f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:37 UTC) from {username='slave'} at 127.0.0.1:35560
I20260927 22:01:37.669734 8652 tablet_service.cc:1514] Processing DeleteTablet for tablet b52d5f888ee74a6eaf44f584d95c5f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:37 UTC) from {username='slave'} at 127.0.0.1:34210
I20260927 22:01:37.670722 8891 tablet_replica.cc:330] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: stopping tablet replica
I20260927 22:01:37.672488 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 8459
I20260927 22:01:37.673205 8891 raft_consensus.cc:2229] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:37.673890 8891 raft_consensus.cc:2258] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:37.675141 8893 tablet_replica.cc:330] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: stopping tablet replica
I20260927 22:01:37.675626 8893 raft_consensus.cc:2229] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:01:37.676431 8893 raft_consensus.cc:2258] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:37.677310 8892 tablet_replica.cc:330] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a: stopping tablet replica
I20260927 22:01:37.677814 8892 raft_consensus.cc:2229] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:37.678380 8892 raft_consensus.cc:2258] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:37.679952 8891 ts_tablet_manager.cc:1905] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:37.679910 8893 ts_tablet_manager.cc:1905] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:37.681568 8892 ts_tablet_manager.cc:1905] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:37.691221 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.676915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35574 (local address 127.3.159.1:43201)
0927 22:01:37.677144 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:37.677151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:37.688240 (+ 11089us) server_negotiation.cc:407] Connection header received
0927 22:01:37.688567 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:37.688576 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:37.688736 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:37.688828 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:37.690263 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:37.690270 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:37.690279 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:37.690300 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:37.690345 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:37.690363 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:37.690490 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:37.690741 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 22:01:37.690822 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:01:37.691524 8893 ts_tablet_manager.cc:1918] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:37.691829 8893 log.cc:1196] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/b52d5f888ee74a6eaf44f584d95c5f11
I20260927 22:01:37.692938 8581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:37.697865 8891 ts_tablet_manager.cc:1918] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:37.697950 8893 ts_tablet_manager.cc:1939] T b52d5f888ee74a6eaf44f584d95c5f11 P f63d690933b1484091484839bbd1c4fa: Deleting consensus metadata
I20260927 22:01:37.698117 8891 log.cc:1196] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/b52d5f888ee74a6eaf44f584d95c5f11
I20260927 22:01:37.698911 8891 ts_tablet_manager.cc:1939] T b52d5f888ee74a6eaf44f584d95c5f11 P 2319893a9cf740caa9e1a79312961478: Deleting consensus metadata
I20260927 22:01:37.700421 8385 catalog_manager.cc:4841] TS f63d690933b1484091484839bbd1c4fa (127.3.159.2:39251): tablet b52d5f888ee74a6eaf44f584d95c5f11 (table test-table [id=2ed7f4e2c3f84f20b0ef485c4e391e84]) successfully deleted
I20260927 22:01:37.701306 8386 catalog_manager.cc:4841] TS 2319893a9cf740caa9e1a79312961478 (127.3.159.3:43347): tablet b52d5f888ee74a6eaf44f584d95c5f11 (table test-table [id=2ed7f4e2c3f84f20b0ef485c4e391e84]) successfully deleted
I20260927 22:01:37.812944 8892 ts_tablet_manager.cc:1918] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:37.813280 8892 log.cc:1196] T b52d5f888ee74a6eaf44f584d95c5f11 P 50ace9f4f8324937bb32cd18fcdc0a2a: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/b52d5f888ee74a6eaf44f584d95c5f11
I20260927 22:01:37.813879 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 8459
W20260927 22:01:37.834216 8387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.159.1:43201 (error 108)
I20260927 22:01:37.834340 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 8590
I20260927 22:01:37.836830 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.836171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:43201 (local address 127.0.0.1:35580)
0927 22:01:37.836481 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:37.836635 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
W20260927 22:01:37.837141 8387 catalog_manager.cc:4584] TS 50ace9f4f8324937bb32cd18fcdc0a2a (127.3.159.1:43201): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b52d5f888ee74a6eaf44f584d95c5f11: Network error: Client connection negotiation failed: client connection to 127.3.159.1:43201: connect: Connection refused (error 111)
I20260927 22:01:37.838295 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.835659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34224 (local address 127.3.159.2:39251)
0927 22:01:37.835895 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:37.835902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:37.835922 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:01:37.836015 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:37.836019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:37.836080 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:37.836216 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:37.837582 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:37.837589 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:37.837594 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:37.837620 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:37.837651 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:37.837664 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:37.837819 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:37.837911 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 22:01:37.837980 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 22:01:37.839054 8712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:37.891259 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.890503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:43201 (local address 127.0.0.1:35590)
0927 22:01:37.890827 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:37.890995 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 22:01:37.893021 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.892482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:43201 (local address 127.0.0.1:35598)
0927 22:01:37.892698 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:37.892822 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 22:01:37.954375 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.953715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:43201 (local address 127.0.0.1:35600)
0927 22:01:37.953999 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:37.954131 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 22:01:37.955911 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.955424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:43201 (local address 127.0.0.1:35616)
0927 22:01:37.955634 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:37.955750 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:01:37.975983 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 8590
I20260927 22:01:37.993729 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 8721
I20260927 22:01:37.997582 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:37.994952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.3.159.3:43347)
0927 22:01:37.995293 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:37.995302 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:37.995329 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:01:37.995400 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:37.995408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:37.995516 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:37.995656 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:37.996791 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:37.996802 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:37.996806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:37.996822 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:37.996857 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:37.996871 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:37.997038 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:37.997098 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 22:01:37.997180 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260927 22:01:37.998462 8843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:38.037153 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:38.036508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:43201 (local address 127.0.0.1:35628)
0927 22:01:38.036816 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:38.036935 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 22:01:38.038357 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:38.037952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:43201 (local address 127.0.0.1:35632)
0927 22:01:38.038160 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:38.038236 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 22:01:38.098544 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 8721
I20260927 22:01:38.117818 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 8369
I20260927 22:01:38.119570 8430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:38.241242 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:38.240284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:43201 (local address 127.0.0.1:35648)
0927 22:01:38.240550 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:38.240728 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:43201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:01:38.241557 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 8369
2026-09-27T22:01:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3380 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 22:01:38.288699 3708 mini_oidc.cc:152] Starting JWKS server
I20260927 22:01:38.291985 3708 webserver.cc:460] Webserver started at https://127.0.0.1:41829/ using document root <none> and password file <none>
I20260927 22:01:38.292359 3708 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:01:38.293216 3708 webserver.cc:460] Webserver started at http://127.0.0.1:45401/ using document root <none> and password file <none>
2026-09-27T22:01:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:38Z Disabled control of system clock
I20260927 22:01:38.327824 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:42675
--webserver_interface=127.3.159.62
--webserver_port=0
--builtin_ntp_servers=127.3.159.20:34105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:42675
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41829/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/testchainca.cert with env {}
W20260927 22:01:38.699723 8905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:38.700106 8905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:38.700273 8905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:38.707712 8905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:38.707894 8905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:38.707998 8905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:38.708097 8905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:38.708184 8905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:01:38.708273 8905 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:01:38.708352 8905 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41829/jwks.json
I20260927 22:01:38.718756 8905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:42675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:42675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41829/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:38.720206 8905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:38.721612 8905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:38.735507 8910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:38.735569 8913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:38.735469 8911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:38.736542 8905 server_base.cc:1034] running on GCE node
I20260927 22:01:38.737555 8905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:38.739250 8905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:38.740532 8905 hybrid_clock.cc:648] HybridClock initialized: now 1790546498740445 us; error 85 us; skew 500 ppm
I20260927 22:01:38.740938 8905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:01:38.743889 8905 webserver.cc:460] Webserver started at http://127.3.159.62:41921/ using document root <none> and password file <none>
I20260927 22:01:38.744756 8905 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:38.744922 8905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:38.745277 8905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:38.748564 8905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "e56504393403435f9e3aee9342545fae"
format_stamp: "Formatted at 2026-09-27 22:01:38 on dist-test-slave-sg4j"
server_key: "5ccf3374174edf649656c2620ea8e7f4"
server_key_iv: "46835e60f55f85dcdc33b9a90f01dcec"
server_key_version: "encryptionkey@0"
I20260927 22:01:38.749634 8905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "e56504393403435f9e3aee9342545fae"
format_stamp: "Formatted at 2026-09-27 22:01:38 on dist-test-slave-sg4j"
server_key: "5ccf3374174edf649656c2620ea8e7f4"
server_key_iv: "46835e60f55f85dcdc33b9a90f01dcec"
server_key_version: "encryptionkey@0"
I20260927 22:01:38.755965 8905 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:01:38.760406 8919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:38.762521 8905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260927 22:01:38.762816 8905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "e56504393403435f9e3aee9342545fae"
format_stamp: "Formatted at 2026-09-27 22:01:38 on dist-test-slave-sg4j"
server_key: "5ccf3374174edf649656c2620ea8e7f4"
server_key_iv: "46835e60f55f85dcdc33b9a90f01dcec"
server_key_version: "encryptionkey@0"
I20260927 22:01:38.763087 8905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:38.786854 8905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:38.787833 8905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:38.788514 8905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:38.812543 8905 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:42675
I20260927 22:01:38.812597 8970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:42675 every 8 connection(s)
I20260927 22:01:38.814101 8905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:38.819113 8971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:38.824885 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 8905
I20260927 22:01:38.825212 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:38.826216 3708 external_mini_cluster.cc:1428] Setting key 76e5195e3d64f54ebc7ce8482482cdde
I20260927 22:01:38.833895 8971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae: Bootstrap starting.
I20260927 22:01:38.838949 8971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:38.840448 8971 log.cc:824] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:38.844033 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:38.828795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56482 (local address 127.3.159.62:42675)
0927 22:01:38.830025 (+ 1230us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:38.830042 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:38.830098 (+ 56us) server_negotiation.cc:407] Connection header received
0927 22:01:38.832170 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:38.832198 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:38.832801 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:38.833215 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:38.834319 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:38.835827 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:38.838144 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:38.839002 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:38.839312 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:38.841187 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:38.841196 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:38.841215 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:38.841987 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:38.842089 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:38.842115 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:38.842274 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:38.842544 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 22:01:38.842907 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":427,"threads_started":1}
I20260927 22:01:38.844527 8971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae: No bootstrap required, opened a new log
I20260927 22:01:38.849062 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:01:38.854216 8971 raft_consensus.cc:348] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56504393403435f9e3aee9342545fae" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42675 } }
I20260927 22:01:38.854785 8971 raft_consensus.cc:374] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:38.854908 8971 raft_consensus.cc:729] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56504393403435f9e3aee9342545fae, State: Initialized, Role: FOLLOWER
I20260927 22:01:38.855844 8971 consensus_queue.cc:260] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56504393403435f9e3aee9342545fae" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42675 } }
I20260927 22:01:38.856164 8971 raft_consensus.cc:388] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:38.856340 8971 raft_consensus.cc:482] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:38.856586 8971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:38.859473 8971 raft_consensus.cc:504] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56504393403435f9e3aee9342545fae" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42675 } }
I20260927 22:01:38.860271 8971 leader_election.cc:302] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e56504393403435f9e3aee9342545fae; no voters:
I20260927 22:01:38.861166 8971 leader_election.cc:288] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:38.861423 8976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:38.862421 8976 raft_consensus.cc:686] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [term 1 LEADER]: Becoming Leader. State: Replica: e56504393403435f9e3aee9342545fae, State: Running, Role: LEADER
I20260927 22:01:38.863307 8976 consensus_queue.cc:237] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56504393403435f9e3aee9342545fae" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42675 } }
I20260927 22:01:38.864486 8971 sys_catalog.cc:564] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:38.867018 8978 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [sys.catalog]: SysCatalogTable state changed. Reason: New leader e56504393403435f9e3aee9342545fae. Latest consensus state: current_term: 1 leader_uuid: "e56504393403435f9e3aee9342545fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56504393403435f9e3aee9342545fae" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42675 } } }
I20260927 22:01:38.867027 8977 sys_catalog.cc:455] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e56504393403435f9e3aee9342545fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e56504393403435f9e3aee9342545fae" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42675 } } }
I20260927 22:01:38.867908 8977 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:38.870267 8978 sys_catalog.cc:458] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:38.871929 8982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:38.878921 8982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:38.892460 8982 catalog_manager.cc:1347] Generated new cluster ID: e24600af5c7b4cb7a36d7e3ed830cd3b
I20260927 22:01:38.892670 8982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:38.912950 8982 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:38.914695 8982 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:38.926069 8982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e56504393403435f9e3aee9342545fae: Generated new TSK 0
I20260927 22:01:38.927403 8982 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:01:38.977713 9001 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 22:01:38.977926 9001 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:01:38.979243 8974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:38.955731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.3.159.62:42675)
0927 22:01:38.956002 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:38.956011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:38.956449 (+ 438us) server_negotiation.cc:407] Connection header received
0927 22:01:38.956851 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:38.956855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:38.956940 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:38.957111 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:01:38.957821 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:38.958597 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:38.960119 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:38.960613 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:38.960759 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:38.961441 (+ 682us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:01:38.978403 (+ 16962us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:01:38.978859 (+ 456us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":118}
W20260927 22:01:38.979390 8974 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 22:01:38.981232 9000 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:01:38.955414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:42675 (local address 127.0.0.1:56488)
0927 22:01:38.956302 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:38.956322 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:38.956571 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:38.957142 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:38.957147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:38.957160 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:01:38.957640 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:38.957650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:38.958747 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:38.958753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:38.959912 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:38.959923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:38.960835 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:38.960842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:38.960981 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:38.961064 (+ 83us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:01:38.978748 (+ 17684us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 22:01:38.979134 (+ 386us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.159.62:42675: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":489,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.159.62:42675: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41829/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 22:01:38.983767 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 8905
I20260927 22:01:38.985859 8966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:39.095901 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 8905
2026-09-27T22:01:39Z chronyd exiting
I20260927 22:01:39.152287 3708 test_util.cc:175] -----------------------------------------------
I20260927 22:01:39.152542 3708 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790546447244210-3708-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (869 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 22:01:39.157759 3708 mini_oidc.cc:152] Starting JWKS server
I20260927 22:01:39.160140 3708 webserver.cc:460] Webserver started at https://127.0.0.1:42685/ using document root <none> and password file <none>
I20260927 22:01:39.160513 3708 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:01:39.161450 3708 webserver.cc:460] Webserver started at http://127.0.0.1:41159/ using document root <none> and password file <none>
2026-09-27T22:01:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:39Z Disabled control of system clock
I20260927 22:01:39.201193 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:33107
--webserver_interface=127.3.159.62
--webserver_port=0
--builtin_ntp_servers=127.3.159.20:40671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:33107
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42685/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/testchainca.cert with env {}
W20260927 22:01:39.602104 9008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:39.602526 9008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:39.602712 9008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:39.610702 9008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:39.610911 9008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:39.611024 9008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:39.611143 9008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:39.611243 9008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:01:39.611583 9008 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:01:39.611716 9008 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42685/jwks.json
I20260927 22:01:39.623015 9008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:40671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:33107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:33107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42685/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:39.624547 9008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:39.626191 9008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:39.641501 9014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:39.641649 9013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:39.641635 9016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:39.644528 9008 server_base.cc:1034] running on GCE node
I20260927 22:01:39.645673 9008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:39.647683 9008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:39.649055 9008 hybrid_clock.cc:648] HybridClock initialized: now 1790546499648889 us; error 142 us; skew 500 ppm
I20260927 22:01:39.649634 9008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:01:39.653105 9008 webserver.cc:460] Webserver started at http://127.3.159.62:36079/ using document root <none> and password file <none>
I20260927 22:01:39.654129 9008 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:39.654297 9008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:39.654712 9008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:39.658556 9008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "54238f198bcd45ab9448b4b7defe8e4a"
format_stamp: "Formatted at 2026-09-27 22:01:39 on dist-test-slave-sg4j"
server_key: "522d52cc2446e39a9d3b8a1008e98fb1"
server_key_iv: "b71b99ddefd6b06f25929e3eeec6faa9"
server_key_version: "encryptionkey@0"
I20260927 22:01:39.659565 9008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "54238f198bcd45ab9448b4b7defe8e4a"
format_stamp: "Formatted at 2026-09-27 22:01:39 on dist-test-slave-sg4j"
server_key: "522d52cc2446e39a9d3b8a1008e98fb1"
server_key_iv: "b71b99ddefd6b06f25929e3eeec6faa9"
server_key_version: "encryptionkey@0"
I20260927 22:01:39.666850 9008 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260927 22:01:39.671883 9022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:39.674194 9008 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 22:01:39.674475 9008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "54238f198bcd45ab9448b4b7defe8e4a"
format_stamp: "Formatted at 2026-09-27 22:01:39 on dist-test-slave-sg4j"
server_key: "522d52cc2446e39a9d3b8a1008e98fb1"
server_key_iv: "b71b99ddefd6b06f25929e3eeec6faa9"
server_key_version: "encryptionkey@0"
I20260927 22:01:39.674764 9008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:39.691401 9008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:39.692500 9008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:39.692858 9008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:39.716172 9008 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:33107
I20260927 22:01:39.716249 9073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:33107 every 8 connection(s)
I20260927 22:01:39.718014 9008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:39.720687 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 9008
I20260927 22:01:39.720956 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:39.721724 3708 external_mini_cluster.cc:1428] Setting key 780778e60e6cc9b0b711a03a22c3a59b
I20260927 22:01:39.723851 9074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:39.738116 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:39.724237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.3.159.62:33107)
0927 22:01:39.725635 (+ 1398us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:39.725665 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:39.725751 (+ 86us) server_negotiation.cc:407] Connection header received
0927 22:01:39.727574 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:39.727601 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:39.728238 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:39.728887 (+ 649us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:39.729626 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:39.731212 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:39.732692 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:39.733611 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:39.733860 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:39.735361 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:39.735370 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:39.735388 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:39.736072 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:39.736166 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:39.736193 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:39.736332 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:39.736601 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 22:01:39.736987 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":454,"threads_started":1}
I20260927 22:01:39.739226 9074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a: Bootstrap starting.
I20260927 22:01:39.743265 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:01:39.743814 9074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:39.745110 9074 log.cc:824] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:39.748960 9074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a: No bootstrap required, opened a new log
I20260927 22:01:39.757465 9074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54238f198bcd45ab9448b4b7defe8e4a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33107 } }
I20260927 22:01:39.758064 9074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:39.758183 9074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54238f198bcd45ab9448b4b7defe8e4a, State: Initialized, Role: FOLLOWER
I20260927 22:01:39.758957 9074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54238f198bcd45ab9448b4b7defe8e4a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33107 } }
I20260927 22:01:39.759292 9074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:39.759547 9074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:39.759774 9074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:39.762912 9074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54238f198bcd45ab9448b4b7defe8e4a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33107 } }
I20260927 22:01:39.763649 9074 leader_election.cc:302] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54238f198bcd45ab9448b4b7defe8e4a; no voters:
I20260927 22:01:39.764482 9074 leader_election.cc:288] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:39.764818 9079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:39.766074 9079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [term 1 LEADER]: Becoming Leader. State: Replica: 54238f198bcd45ab9448b4b7defe8e4a, State: Running, Role: LEADER
I20260927 22:01:39.766834 9079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54238f198bcd45ab9448b4b7defe8e4a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33107 } }
I20260927 22:01:39.767486 9074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:39.770068 9080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54238f198bcd45ab9448b4b7defe8e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54238f198bcd45ab9448b4b7defe8e4a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33107 } } }
I20260927 22:01:39.770588 9080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:39.770951 9081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54238f198bcd45ab9448b4b7defe8e4a. Latest consensus state: current_term: 1 leader_uuid: "54238f198bcd45ab9448b4b7defe8e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54238f198bcd45ab9448b4b7defe8e4a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33107 } } }
I20260927 22:01:39.771347 9081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:39.774587 9084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:39.782379 9084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:39.794194 9084 catalog_manager.cc:1347] Generated new cluster ID: accc8ee8bc694dc898d8e948878f2d83
I20260927 22:01:39.794399 9084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:39.807265 9084 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:39.808912 9084 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:39.825601 9084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54238f198bcd45ab9448b4b7defe8e4a: Generated new TSK 0
I20260927 22:01:39.826982 9084 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 22:01:39.874776 9104 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 22:01:39.874957 9104 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:01:39.876307 9077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:39.853502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.3.159.62:33107)
0927 22:01:39.853850 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:39.853860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:39.854156 (+ 296us) server_negotiation.cc:407] Connection header received
0927 22:01:39.854411 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:39.854418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:39.854530 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:39.854753 (+ 223us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:01:39.855361 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:39.856501 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:39.858425 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:39.859191 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:39.859374 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:39.860022 (+ 648us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:01:39.875505 (+ 15483us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42685/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:01:39.875939 (+ 434us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42685/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":186}
W20260927 22:01:39.876466 9077 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42685/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 22:01:39.876629 9103 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:01:39.853247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33107 (local address 127.0.0.1:39124)
0927 22:01:39.854022 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:39.854041 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:39.854198 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:39.854832 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:39.854837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:39.854849 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:01:39.855166 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:39.855175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:39.856733 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:39.856738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:39.858229 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:39.858244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:39.859440 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:39.859445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:39.859609 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:39.859654 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:01:39.875868 (+ 16214us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42685/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 22:01:39.876187 (+ 319us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.159.62:33107: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42685/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":457,"threads_started":1}
I20260927 22:01:39.878366 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 9008
I20260927 22:01:39.880287 9069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:39.987921 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 9008
2026-09-27T22:01:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (891 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 22:01:40.049535 3708 mini_oidc.cc:152] Starting JWKS server
I20260927 22:01:40.051740 3708 webserver.cc:460] Webserver started at https://127.0.0.1:36463/ using document root <none> and password file <none>
I20260927 22:01:40.051981 3708 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 22:01:40.052771 3708 webserver.cc:460] Webserver started at http://127.0.0.1:45187/ using document root <none> and password file <none>
2026-09-27T22:01:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:40Z Disabled control of system clock
I20260927 22:01:40.088477 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:46779
--webserver_interface=127.3.159.62
--webserver_port=0
--builtin_ntp_servers=127.3.159.20:46229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:46779
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36463/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:01:40.470911 9111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:40.471367 9111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:40.471566 9111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:40.481483 9111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:40.481871 9111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:40.481981 9111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:40.482079 9111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:40.482154 9111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:01:40.482273 9111 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:01:40.482336 9111 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36463/jwks.json
I20260927 22:01:40.494129 9111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:46229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:46779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36463/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:40.495630 9111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:40.497720 9111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:40.511368 9117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:40.511479 9116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:40.511562 9119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:40.514046 9111 server_base.cc:1034] running on GCE node
I20260927 22:01:40.515363 9111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:40.517661 9111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:40.519165 9111 hybrid_clock.cc:648] HybridClock initialized: now 1790546500519070 us; error 95 us; skew 500 ppm
I20260927 22:01:40.519984 9111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:01:40.523427 9111 webserver.cc:460] Webserver started at http://127.3.159.62:35245/ using document root <none> and password file <none>
I20260927 22:01:40.524323 9111 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:40.524482 9111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:40.525074 9111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:40.528753 9111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "b559f85044d64f8faa5efb677865b80b"
format_stamp: "Formatted at 2026-09-27 22:01:40 on dist-test-slave-sg4j"
server_key: "2b25f59ebd66e9b013a8a0ff1a956a5f"
server_key_iv: "5cc79c7edca9129b6782637046a8d181"
server_key_version: "encryptionkey@0"
I20260927 22:01:40.529759 9111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "b559f85044d64f8faa5efb677865b80b"
format_stamp: "Formatted at 2026-09-27 22:01:40 on dist-test-slave-sg4j"
server_key: "2b25f59ebd66e9b013a8a0ff1a956a5f"
server_key_iv: "5cc79c7edca9129b6782637046a8d181"
server_key_version: "encryptionkey@0"
I20260927 22:01:40.536367 9111 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:01:40.542148 9125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:40.544248 9111 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260927 22:01:40.544548 9111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "b559f85044d64f8faa5efb677865b80b"
format_stamp: "Formatted at 2026-09-27 22:01:40 on dist-test-slave-sg4j"
server_key: "2b25f59ebd66e9b013a8a0ff1a956a5f"
server_key_iv: "5cc79c7edca9129b6782637046a8d181"
server_key_version: "encryptionkey@0"
I20260927 22:01:40.544832 9111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:40.573840 9111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:40.574837 9111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:40.575196 9111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:40.598608 9111 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:46779
I20260927 22:01:40.598654 9176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:46779 every 8 connection(s)
I20260927 22:01:40.600082 9111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:40.605145 9177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:40.606812 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 9111
I20260927 22:01:40.607060 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:40.607738 3708 external_mini_cluster.cc:1428] Setting key 010fdfb4974cc39a39828ad530bf4075
I20260927 22:01:40.620846 9177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b: Bootstrap starting.
I20260927 22:01:40.623310 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:40.610293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.3.159.62:46779)
0927 22:01:40.611523 (+ 1230us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:40.611535 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:40.611580 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:01:40.613351 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:40.613441 (+ 90us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:40.614048 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:40.614585 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:40.615276 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:40.616691 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:40.618181 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:40.618983 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:40.619205 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:40.620525 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:40.620533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:40.620551 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:40.621267 (+ 716us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:40.621363 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:40.621442 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:40.621583 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:40.621821 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 22:01:40.622188 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":462,"threads_started":1}
I20260927 22:01:40.625437 9177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:40.626991 9177 log.cc:824] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:40.627987 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:01:40.630764 9177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b: No bootstrap required, opened a new log
I20260927 22:01:40.639174 9177 raft_consensus.cc:348] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b559f85044d64f8faa5efb677865b80b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46779 } }
I20260927 22:01:40.639730 9177 raft_consensus.cc:374] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:40.639850 9177 raft_consensus.cc:729] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b559f85044d64f8faa5efb677865b80b, State: Initialized, Role: FOLLOWER
I20260927 22:01:40.640702 9177 consensus_queue.cc:260] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b559f85044d64f8faa5efb677865b80b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46779 } }
I20260927 22:01:40.641004 9177 raft_consensus.cc:388] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:40.641176 9177 raft_consensus.cc:482] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:40.641413 9177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:40.644066 9177 raft_consensus.cc:504] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b559f85044d64f8faa5efb677865b80b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46779 } }
I20260927 22:01:40.644757 9177 leader_election.cc:302] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b559f85044d64f8faa5efb677865b80b; no voters:
I20260927 22:01:40.645615 9177 leader_election.cc:288] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:40.645900 9182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:40.646970 9182 raft_consensus.cc:686] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [term 1 LEADER]: Becoming Leader. State: Replica: b559f85044d64f8faa5efb677865b80b, State: Running, Role: LEADER
I20260927 22:01:40.647629 9182 consensus_queue.cc:237] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b559f85044d64f8faa5efb677865b80b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46779 } }
I20260927 22:01:40.648491 9177 sys_catalog.cc:564] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:40.651017 9184 sys_catalog.cc:455] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b559f85044d64f8faa5efb677865b80b. Latest consensus state: current_term: 1 leader_uuid: "b559f85044d64f8faa5efb677865b80b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b559f85044d64f8faa5efb677865b80b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46779 } } }
I20260927 22:01:40.651499 9184 sys_catalog.cc:458] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:40.652792 9183 sys_catalog.cc:455] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b559f85044d64f8faa5efb677865b80b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b559f85044d64f8faa5efb677865b80b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 46779 } } }
I20260927 22:01:40.653520 9183 sys_catalog.cc:458] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:40.659454 9187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:40.666304 9187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:40.679754 9187 catalog_manager.cc:1347] Generated new cluster ID: 350826c529a44600963c71c7cf035777
I20260927 22:01:40.679953 9187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:40.701709 9187 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:40.703495 9187 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:40.711333 9187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b559f85044d64f8faa5efb677865b80b: Generated new TSK 0
I20260927 22:01:40.712513 9187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:01:40.749146 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:40.737235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42718 (local address 127.3.159.62:46779)
0927 22:01:40.737584 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:40.737591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:40.738150 (+ 559us) server_negotiation.cc:407] Connection header received
0927 22:01:40.738545 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:40.738551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:40.738652 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:40.738883 (+ 231us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:01:40.739637 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:40.740571 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:40.743194 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:40.743885 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:40.744082 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:40.744967 (+ 885us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:01:40.748190 (+ 3223us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36463/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:01:40.748628 (+ 438us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36463/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":162}
W20260927 22:01:40.749325 9180 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42718: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36463/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 22:01:40.751668 9206 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:01:40.736963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:46779 (local address 127.0.0.1:42718)
0927 22:01:40.737933 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:40.737963 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:40.738208 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:40.738936 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:40.738943 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:40.738959 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:01:40.739387 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:40.739402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:40.740802 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:40.740808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:40.742877 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:40.742897 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:40.744217 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:40.744225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:40.744466 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:40.744530 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:01:40.748640 (+ 4110us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36463/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 22:01:40.751152 (+ 2512us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.159.62:46779: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36463/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":1966,"thread_start_us":498,"threads_started":1}
I20260927 22:01:40.753302 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 9111
I20260927 22:01:40.754961 9172 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 22:01:40.761806 9207 webserver.cc:533] Webserver: sslize failed:
E20260927 22:01:40.761932 9207 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 22:01:40.854532 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 9111
2026-09-27T22:01:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (867 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T22:01:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:40Z Disabled control of system clock
I20260927 22:01:40.950408 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:33799
--webserver_interface=127.3.159.62
--webserver_port=0
--builtin_ntp_servers=127.3.159.20:40583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:33799
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/testchainca.cert with env {}
W20260927 22:01:41.354990 9212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:41.355413 9212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:41.355522 9212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:41.364080 9212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:41.364289 9212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:41.364359 9212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:41.364442 9212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:41.364514 9212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 22:01:41.364583 9212 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 22:01:41.364617 9212 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 22:01:41.376953 9212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:40583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:33799
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:41.379106 9212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:41.380869 9212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:41.396543 9217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:41.396739 9218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:41.397253 9220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:41.398313 9212 server_base.cc:1034] running on GCE node
I20260927 22:01:41.399372 9212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:41.401331 9212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:41.402679 9212 hybrid_clock.cc:648] HybridClock initialized: now 1790546501402593 us; error 78 us; skew 500 ppm
I20260927 22:01:41.403231 9212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:01:41.406647 9212 webserver.cc:460] Webserver started at http://127.3.159.62:46623/ using document root <none> and password file <none>
I20260927 22:01:41.407629 9212 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:41.407799 9212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:41.408200 9212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:41.411974 9212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "bd94a402d46a4374814aa2de7233386e"
format_stamp: "Formatted at 2026-09-27 22:01:41 on dist-test-slave-sg4j"
server_key: "3ffead9690b918011bb3db947f3455d6"
server_key_iv: "3f1acf89c01d3cd9e63707ae4593f0ac"
server_key_version: "encryptionkey@0"
I20260927 22:01:41.412971 9212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "bd94a402d46a4374814aa2de7233386e"
format_stamp: "Formatted at 2026-09-27 22:01:41 on dist-test-slave-sg4j"
server_key: "3ffead9690b918011bb3db947f3455d6"
server_key_iv: "3f1acf89c01d3cd9e63707ae4593f0ac"
server_key_version: "encryptionkey@0"
I20260927 22:01:41.419879 9212 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260927 22:01:41.425179 9226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:41.427317 9212 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:01:41.427627 9212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "bd94a402d46a4374814aa2de7233386e"
format_stamp: "Formatted at 2026-09-27 22:01:41 on dist-test-slave-sg4j"
server_key: "3ffead9690b918011bb3db947f3455d6"
server_key_iv: "3f1acf89c01d3cd9e63707ae4593f0ac"
server_key_version: "encryptionkey@0"
I20260927 22:01:41.427934 9212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:41.460753 9212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:41.461958 9212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:41.462332 9212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:41.486218 9212 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:33799
I20260927 22:01:41.486241 9277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:33799 every 8 connection(s)
I20260927 22:01:41.488214 9212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:41.493028 9278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:41.498066 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 9212
I20260927 22:01:41.498355 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:41.499142 3708 external_mini_cluster.cc:1428] Setting key 15d487bcba93322b3199f1be551e7ffc
I20260927 22:01:41.506024 9278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e: Bootstrap starting.
I20260927 22:01:41.510475 9278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:41.511919 9278 log.cc:824] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:41.516350 9278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e: No bootstrap required, opened a new log
I20260927 22:01:41.517853 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:41.501790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53240 (local address 127.3.159.62:33799)
0927 22:01:41.503288 (+ 1498us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:41.503304 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:41.503372 (+ 68us) server_negotiation.cc:407] Connection header received
0927 22:01:41.505566 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:41.505599 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:41.506317 (+ 718us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:41.506791 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:41.507622 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:41.509350 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:41.511605 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:41.512660 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:41.512936 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:41.514402 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:41.514413 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:41.514435 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:01:41.515157 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:41.515271 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:01:41.515298 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:41.515431 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:41.515778 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 22:01:41.516269 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":515,"threads_started":1}
I20260927 22:01:41.523502 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:01:41.526643 9278 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd94a402d46a4374814aa2de7233386e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33799 } }
I20260927 22:01:41.527169 9278 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:41.527315 9278 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd94a402d46a4374814aa2de7233386e, State: Initialized, Role: FOLLOWER
I20260927 22:01:41.528117 9278 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd94a402d46a4374814aa2de7233386e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33799 } }
I20260927 22:01:41.528427 9278 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:41.528604 9278 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:41.528805 9278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:41.531870 9278 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd94a402d46a4374814aa2de7233386e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33799 } }
I20260927 22:01:41.532557 9278 leader_election.cc:302] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd94a402d46a4374814aa2de7233386e; no voters:
I20260927 22:01:41.533313 9278 leader_election.cc:288] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:41.533667 9283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:41.534694 9283 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [term 1 LEADER]: Becoming Leader. State: Replica: bd94a402d46a4374814aa2de7233386e, State: Running, Role: LEADER
I20260927 22:01:41.535360 9283 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd94a402d46a4374814aa2de7233386e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33799 } }
I20260927 22:01:41.536023 9278 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:41.539983 9284 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd94a402d46a4374814aa2de7233386e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd94a402d46a4374814aa2de7233386e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33799 } } }
I20260927 22:01:41.540494 9284 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:41.540112 9285 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd94a402d46a4374814aa2de7233386e. Latest consensus state: current_term: 1 leader_uuid: "bd94a402d46a4374814aa2de7233386e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd94a402d46a4374814aa2de7233386e" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33799 } } }
I20260927 22:01:41.540634 9285 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:41.542822 9289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:41.549165 9289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:41.561090 9289 catalog_manager.cc:1347] Generated new cluster ID: 61eb2435fd324316a1af8deb6d4f0060
I20260927 22:01:41.561307 9289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:41.580868 9289 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:41.582629 9289 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:41.598392 9289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd94a402d46a4374814aa2de7233386e: Generated new TSK 0
I20260927 22:01:41.599677 9289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:01:42.080053 9281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:41.621017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.3.159.62:33799)
0927 22:01:41.621287 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:41.621295 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:41.621902 (+ 607us) server_negotiation.cc:407] Connection header received
0927 22:01:41.622190 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:41.622197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:41.622306 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:41.622531 (+ 225us) server_negotiation.cc:226] Negotiated authn=JWT
0927 22:01:41.623110 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:41.624305 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:41.626111 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:41.626918 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:41.627084 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:41.627692 (+ 608us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 22:01:42.079331 (+451639us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:01:42.079778 (+ 447us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53244: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 22:01:42.080255 9281 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53244: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 22:01:42.080425 9307 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:01:41.620788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33799 (local address 127.0.0.1:53244)
0927 22:01:41.621764 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:41.621785 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:41.621961 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:41.622554 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:41.622558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:41.622569 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 22:01:41.622931 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:41.622940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:41.624513 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:41.624519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:41.625925 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:41.625939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:41.627149 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:41.627155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:41.627299 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:41.627352 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 22:01:42.079725 (+452373us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 22:01:42.080026 (+ 301us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.159.62:33799: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":562,"threads_started":1}
I20260927 22:01:42.082057 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 9212
I20260927 22:01:42.084162 9273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:42.195189 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 9212
2026-09-27T22:01:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1321 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (516 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1631 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:01:44 dist-test-slave-sg4j krb5kdc[9323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:01:44 dist-test-slave-sg4j krb5kdc[9323](info): set up 2 sockets
Sep 27 22:01:44 dist-test-slave-sg4j krb5kdc[9323](info): commencing operation
krb5kdc: starting...
W20260927 22:01:45.482765 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:01:45 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:01:45.737942 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:34529
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:34529
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:46.119230 9336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:46.119623 9336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:46.119726 9336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:46.127868 9336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:46.128021 9336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:46.128075 9336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:46.128131 9336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:46.128182 9336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:01:46.139118 9336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:34529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:34529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:46.140796 9336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:46.142561 9336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:46.158149 9344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:46.158519 9342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:46.158914 9341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:46.159608 9336 server_base.cc:1034] running on GCE node
I20260927 22:01:46.160710 9336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:46.162736 9336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:46.164052 9336 hybrid_clock.cc:648] HybridClock initialized: now 1790546506163988 us; error 50 us; skew 500 ppm
I20260927 22:01:46.164563 9336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:46 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546506, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:46.168881 9336 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:01:46.171154 9336 webserver.cc:460] Webserver started at http://127.3.159.62:35807/ using document root <none> and password file <none>
I20260927 22:01:46.172138 9336 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:46.172317 9336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:46.172725 9336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:46.176573 9336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "2140e4eb577745cc9e5eece79f4f0b93"
format_stamp: "Formatted at 2026-09-27 22:01:46 on dist-test-slave-sg4j"
server_key: "9ea15cd77c4c885296d0c85972526a05"
server_key_iv: "2be0e29a286f758266c17f13e2235598"
server_key_version: "encryptionkey@0"
I20260927 22:01:46.177625 9336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "2140e4eb577745cc9e5eece79f4f0b93"
format_stamp: "Formatted at 2026-09-27 22:01:46 on dist-test-slave-sg4j"
server_key: "9ea15cd77c4c885296d0c85972526a05"
server_key_iv: "2be0e29a286f758266c17f13e2235598"
server_key_version: "encryptionkey@0"
I20260927 22:01:46.184746 9336 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260927 22:01:46.190440 9351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:46.192636 9336 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260927 22:01:46.192945 9336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "2140e4eb577745cc9e5eece79f4f0b93"
format_stamp: "Formatted at 2026-09-27 22:01:46 on dist-test-slave-sg4j"
server_key: "9ea15cd77c4c885296d0c85972526a05"
server_key_iv: "2be0e29a286f758266c17f13e2235598"
server_key_version: "encryptionkey@0"
I20260927 22:01:46.193235 9336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:46.229626 9336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:46.235728 9336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:46.236123 9336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:46.260535 9336 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:34529
I20260927 22:01:46.260617 9402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:34529 every 8 connection(s)
I20260927 22:01:46.262444 9336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:46.265905 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 9336
I20260927 22:01:46.266243 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:46.266947 3708 external_mini_cluster.cc:1428] Setting key b48b76fd5666a278bcfae2735878402f
I20260927 22:01:46.267073 9403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:46.282742 9403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93: Bootstrap starting.
Sep 27 22:01:46 dist-test-slave-sg4j krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:46.287467 9403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:46.288858 9403 log.cc:824] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:46.293022 9403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93: No bootstrap required, opened a new log
I20260927 22:01:46.297066 9406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:46.269463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49860 (local address 127.3.159.62:34529)
0927 22:01:46.270551 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:46.270570 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:46.270626 (+ 56us) server_negotiation.cc:407] Connection header received
0927 22:01:46.272609 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:46.272642 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:46.273231 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:46.273755 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:46.275517 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:46.277490 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:46.279011 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:46.280202 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:46.280447 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:46.286466 (+ 6019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:46.286482 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:46.286506 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:46.286569 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:46.291381 (+ 4812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:46.292479 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:46.292489 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:46.292499 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:46.292748 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:46.293279 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:46.293284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:46.293288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:46.294156 (+ 868us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:46.294500 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:46.294917 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 22:01:46.295454 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":468,"threads_started":1}
I20260927 22:01:46.303458 9403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2140e4eb577745cc9e5eece79f4f0b93" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34529 } }
I20260927 22:01:46.303879 9403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:46.303983 9403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2140e4eb577745cc9e5eece79f4f0b93, State: Initialized, Role: FOLLOWER
I20260927 22:01:46.304687 9403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2140e4eb577745cc9e5eece79f4f0b93" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34529 } }
I20260927 22:01:46.304934 9403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:46.305058 9403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:46.305240 9403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:46.308507 9403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2140e4eb577745cc9e5eece79f4f0b93" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34529 } }
I20260927 22:01:46.309082 9403 leader_election.cc:302] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2140e4eb577745cc9e5eece79f4f0b93; no voters:
I20260927 22:01:46.309914 9403 leader_election.cc:288] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:46.310125 9408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:46.311137 9408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [term 1 LEADER]: Becoming Leader. State: Replica: 2140e4eb577745cc9e5eece79f4f0b93, State: Running, Role: LEADER
I20260927 22:01:46.312116 9408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2140e4eb577745cc9e5eece79f4f0b93" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34529 } }
I20260927 22:01:46.312767 9403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:46.316756 9409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2140e4eb577745cc9e5eece79f4f0b93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2140e4eb577745cc9e5eece79f4f0b93" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34529 } } }
I20260927 22:01:46.317257 9410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2140e4eb577745cc9e5eece79f4f0b93. Latest consensus state: current_term: 1 leader_uuid: "2140e4eb577745cc9e5eece79f4f0b93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2140e4eb577745cc9e5eece79f4f0b93" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34529 } } }
I20260927 22:01:46.317677 9409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:46.317737 9410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:46.318991 9417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:46.326632 9417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:46.339847 9417 catalog_manager.cc:1347] Generated new cluster ID: faedc48f8ef54aa28fbe320cee8b5bb5
I20260927 22:01:46.340000 9417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:46.361266 9417 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:46.363659 9417 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:46.376657 9417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2140e4eb577745cc9e5eece79f4f0b93: Generated new TSK 0
I20260927 22:01:46.377823 9417 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:01:46.437944 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34529
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:46.806658 9429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:46.806977 9429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:46.807140 9429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:46.815732 9429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:46.815936 9429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:46.816139 9429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:01:46.827011 9429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:46.828789 9429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:46.830544 9429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:46.846486 9435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:46.846619 9437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:46.846684 9434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:46.847712 9429 server_base.cc:1034] running on GCE node
I20260927 22:01:46.848568 9429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:46.850008 9429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:46.851356 9429 hybrid_clock.cc:648] HybridClock initialized: now 1790546506851261 us; error 94 us; skew 500 ppm
I20260927 22:01:46.851776 9429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:46 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546506, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:46.856047 9429 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:01:46.857967 9429 webserver.cc:460] Webserver started at http://127.3.159.1:45619/ using document root <none> and password file <none>
I20260927 22:01:46.858724 9429 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:46.858841 9429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:46.859107 9429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:46.862275 9429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "4c0dfdf23e4d40b982bcc2246aef3626"
format_stamp: "Formatted at 2026-09-27 22:01:46 on dist-test-slave-sg4j"
server_key: "690ac08f7cdea6cdaa6943e2571abeca"
server_key_iv: "83fdc5cb9b1f765387bacc2840ef8f9a"
server_key_version: "encryptionkey@0"
I20260927 22:01:46.863067 9429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "4c0dfdf23e4d40b982bcc2246aef3626"
format_stamp: "Formatted at 2026-09-27 22:01:46 on dist-test-slave-sg4j"
server_key: "690ac08f7cdea6cdaa6943e2571abeca"
server_key_iv: "83fdc5cb9b1f765387bacc2840ef8f9a"
server_key_version: "encryptionkey@0"
I20260927 22:01:46.869314 9429 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 22:01:46.873699 9444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:46.875638 9429 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:01:46.875902 9429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "4c0dfdf23e4d40b982bcc2246aef3626"
format_stamp: "Formatted at 2026-09-27 22:01:46 on dist-test-slave-sg4j"
server_key: "690ac08f7cdea6cdaa6943e2571abeca"
server_key_iv: "83fdc5cb9b1f765387bacc2840ef8f9a"
server_key_version: "encryptionkey@0"
I20260927 22:01:46.876215 9429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:46.892344 9429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:46.896904 9429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:46.897220 9429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:46.898492 9429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:46.900048 9429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:46.900163 9429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:46.900321 9429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:46.900409 9429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:46.948638 9429 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:45451
I20260927 22:01:46.948705 9556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:45451 every 8 connection(s)
I20260927 22:01:46.950374 9429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:01:46.953781 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 9429
I20260927 22:01:46.954043 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:01:46.954883 3708 external_mini_cluster.cc:1428] Setting key 4320eaa556f48ce7804369c87d3094e0
Sep 27 22:01:46 dist-test-slave-sg4j krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546506, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:46.979745 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:46.954106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:48715 (local address 127.3.159.62:34529)
0927 22:01:46.954714 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:46.954722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:46.956208 (+ 1486us) server_negotiation.cc:407] Connection header received
0927 22:01:46.958881 (+ 2673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:46.958890 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:46.959003 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:46.959177 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:46.961552 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:46.962822 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:46.965703 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:46.966197 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:46.966352 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:46.972336 (+ 5984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:46.972342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:46.972346 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:46.972385 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:46.975294 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:46.976486 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:46.976493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:46.976496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:46.976690 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:46.977441 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:46.977451 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:46.977455 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:46.977815 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:46.977959 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:46.979103 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0927 22:01:46.979335 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":302,"threads_started":1}
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:01:46.981262 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:46.954922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:34529 (local address 127.3.159.1:48715)
0927 22:01:46.955980 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:46.956044 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:46.958524 (+ 2480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:46.959591 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:46.959606 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:46.960346 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:46.961227 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:46.961252 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:46.963059 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:46.963064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:46.965337 (+ 2273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:46.965462 (+ 125us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:46.966554 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:46.966564 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:46.966996 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:46.968508 (+ 1512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:46.968552 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:46.971997 (+ 3445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:46.975524 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:46.975543 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:46.975560 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:46.976256 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:46.976850 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:46.976857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:46.976863 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:46.977173 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:46.977977 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:46.977988 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:46.978566 (+ 578us) client_negotiation.cc:769] Sending connection context
0927 22:01:46.978921 (+ 355us) client_negotiation.cc:240] Negotiation successful
0927 22:01:46.979602 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":423,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:01:46.983775 9557 heartbeater.cc:347] Connected to a master server at 127.3.159.62:34529
I20260927 22:01:46.984201 9557 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:46.985081 9557 heartbeater.cc:511] Master 127.3.159.62:34529 requested a full tablet report, sending...
I20260927 22:01:46.988286 9368 ts_manager.cc:194] Registered new tserver with Master: 4c0dfdf23e4d40b982bcc2246aef3626 (127.3.159.1:45451)
I20260927 22:01:46.990839 9368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:48715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:01:47.017069 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34529
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:47.426781 9563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:47.427220 9563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:47.427417 9563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:47.435364 9563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:47.435577 9563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:47.435773 9563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:01:47.447324 9563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:47.449069 9563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:47.450752 9563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:47.466537 9568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:47.466552 9571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:47.466640 9569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:47.467630 9563 server_base.cc:1034] running on GCE node
I20260927 22:01:47.468552 9563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:47.469988 9563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:47.471256 9563 hybrid_clock.cc:648] HybridClock initialized: now 1790546507471207 us; error 44 us; skew 500 ppm
I20260927 22:01:47.471685 9563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:47 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:47.475839 9563 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:01:47.477972 9563 webserver.cc:460] Webserver started at http://127.3.159.2:45679/ using document root <none> and password file <none>
I20260927 22:01:47.478880 9563 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:47.479043 9563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:47.479419 9563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:47.483282 9563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "25cb3c41a967452586556d12f4842e79"
format_stamp: "Formatted at 2026-09-27 22:01:47 on dist-test-slave-sg4j"
server_key: "283c56f736b2b605109ba5067b06a7da"
server_key_iv: "b33c530f1d883f70e5753d75a6f790ad"
server_key_version: "encryptionkey@0"
I20260927 22:01:47.484212 9563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "25cb3c41a967452586556d12f4842e79"
format_stamp: "Formatted at 2026-09-27 22:01:47 on dist-test-slave-sg4j"
server_key: "283c56f736b2b605109ba5067b06a7da"
server_key_iv: "b33c530f1d883f70e5753d75a6f790ad"
server_key_version: "encryptionkey@0"
I20260927 22:01:47.491024 9563 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260927 22:01:47.495865 9578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:47.497824 9563 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 22:01:47.498042 9563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "25cb3c41a967452586556d12f4842e79"
format_stamp: "Formatted at 2026-09-27 22:01:47 on dist-test-slave-sg4j"
server_key: "283c56f736b2b605109ba5067b06a7da"
server_key_iv: "b33c530f1d883f70e5753d75a6f790ad"
server_key_version: "encryptionkey@0"
I20260927 22:01:47.498355 9563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:47.518451 9563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:47.524346 9563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:47.524742 9563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:47.526119 9563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:47.527906 9563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:47.528028 9563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:47.528218 9563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:47.528301 9563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:47.571684 9563 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:43959
I20260927 22:01:47.571837 9690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:43959 every 8 connection(s)
I20260927 22:01:47.573226 9563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:01:47.575057 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 9563
I20260927 22:01:47.575275 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:01:47.575953 3708 external_mini_cluster.cc:1428] Setting key 02167cdd1c989c2f3ab18f2c512c8df0
Sep 27 22:01:47 dist-test-slave-sg4j krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
I20260927 22:01:47.600725 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:47.577355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:35479 (local address 127.3.159.62:34529)
0927 22:01:47.578153 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:47.578163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:47.579317 (+ 1154us) server_negotiation.cc:407] Connection header received
0927 22:01:47.581810 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:47.581817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:47.581935 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:47.582086 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:47.584213 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:47.585010 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:47.587041 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:47.587542 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:47.587695 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:47.592424 (+ 4729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:47.592435 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:47.592444 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:47.592511 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:47.596342 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:47.597341 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:47.597349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:47.597354 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:47.597539 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:47.598179 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:47.598184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:47.598188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:47.598627 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:47.598773 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:47.599890 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0927 22:01:47.600153 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":304,"threads_started":1}
I20260927 22:01:47.601930 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:47.578059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:34529 (local address 127.3.159.2:35479)
0927 22:01:47.579105 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:47.579166 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:47.581262 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:47.582431 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:47.582443 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:47.583163 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:47.583969 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:47.583994 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:47.585202 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:47.585212 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:47.586821 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:47.586839 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:47.587829 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:47.587837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:47.588127 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:47.589031 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:47.589064 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:47.592043 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:47.596574 (+ 4531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:47.596583 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:47.596603 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:47.597149 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:47.597741 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:47.597746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:47.597749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:47.597984 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:47.598781 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:47.598791 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:47.599375 (+ 584us) client_negotiation.cc:769] Sending connection context
0927 22:01:47.599765 (+ 390us) client_negotiation.cc:240] Negotiation successful
0927 22:01:47.600349 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":401,"threads_started":1}
I20260927 22:01:47.604601 9691 heartbeater.cc:347] Connected to a master server at 127.3.159.62:34529
I20260927 22:01:47.605052 9691 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:47.606245 9691 heartbeater.cc:511] Master 127.3.159.62:34529 requested a full tablet report, sending...
I20260927 22:01:47.608811 9368 ts_manager.cc:194] Registered new tserver with Master: 25cb3c41a967452586556d12f4842e79 (127.3.159.2:43959)
I20260927 22:01:47.610219 9368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:35479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:01:47.629794 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34529
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:47.995042 9557 heartbeater.cc:503] Master 127.3.159.62:34529 was elected leader, sending a full tablet report...
W20260927 22:01:48.029202 9697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:48.029727 9697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:48.029911 9697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:48.038367 9697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:48.038568 9697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:48.038771 9697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:01:48.050699 9697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:48.052515 9697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:48.054235 9697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:48.070613 9702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:48.070704 9703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:48.070642 9705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:48.075459 9697 server_base.cc:1034] running on GCE node
I20260927 22:01:48.076306 9697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:48.077740 9697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:48.079411 9697 hybrid_clock.cc:648] HybridClock initialized: now 1790546508079295 us; error 92 us; skew 500 ppm
I20260927 22:01:48.079885 9697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:48.084383 9697 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:01:48.086544 9697 webserver.cc:460] Webserver started at http://127.3.159.3:35467/ using document root <none> and password file <none>
I20260927 22:01:48.087409 9697 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:48.087580 9697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:48.087914 9697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:48.091225 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "e3f9a255a7c3475f8c70ccbadb8d2af9"
format_stamp: "Formatted at 2026-09-27 22:01:48 on dist-test-slave-sg4j"
server_key: "959415332ee2b0e25ae19993be72ab4a"
server_key_iv: "09fcb96844f7f1ceca68462976e6c340"
server_key_version: "encryptionkey@0"
I20260927 22:01:48.092154 9697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "e3f9a255a7c3475f8c70ccbadb8d2af9"
format_stamp: "Formatted at 2026-09-27 22:01:48 on dist-test-slave-sg4j"
server_key: "959415332ee2b0e25ae19993be72ab4a"
server_key_iv: "09fcb96844f7f1ceca68462976e6c340"
server_key_version: "encryptionkey@0"
I20260927 22:01:48.098721 9697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 22:01:48.104187 9712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:48.106389 9697 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 22:01:48.106671 9697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "e3f9a255a7c3475f8c70ccbadb8d2af9"
format_stamp: "Formatted at 2026-09-27 22:01:48 on dist-test-slave-sg4j"
server_key: "959415332ee2b0e25ae19993be72ab4a"
server_key_iv: "09fcb96844f7f1ceca68462976e6c340"
server_key_version: "encryptionkey@0"
I20260927 22:01:48.106969 9697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:48.128849 9697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:48.135005 9697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:48.135434 9697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:48.136902 9697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:48.139032 9697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:48.139216 9697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:48.139439 9697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:48.139530 9697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:48.187518 9697 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:32947
I20260927 22:01:48.187582 9824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:32947 every 8 connection(s)
I20260927 22:01:48.189258 9697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:01:48.197618 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 9697
I20260927 22:01:48.197860 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:01:48.198386 3708 external_mini_cluster.cc:1428] Setting key bfbe3f1904c89ac870cbb3b994588160
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:01:48.218129 9827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:48.193109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:56457 (local address 127.3.159.62:34529)
0927 22:01:48.193661 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:48.193666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:48.195083 (+ 1417us) server_negotiation.cc:407] Connection header received
0927 22:01:48.197634 (+ 2551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:48.197641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:48.197857 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:48.198043 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:48.200837 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.201941 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.204111 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.204622 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.204761 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:48.209696 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:48.209702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:48.209705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:48.209749 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:48.213837 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.214976 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.214980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.214983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.215106 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.215845 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.215852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.215856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.216243 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:48.216425 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:48.217339 (+ 914us) server_negotiation.cc:299] Negotiation successful
0927 22:01:48.217659 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":318,"threads_started":1}
I20260927 22:01:48.219424 9828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:48.193800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:34529 (local address 127.3.159.3:56457)
0927 22:01:48.194826 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:48.194882 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:48.197008 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:48.198467 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:48.198480 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:48.199537 (+ 1057us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:48.200560 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:48.200590 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.202229 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.202239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:48.203881 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:48.203902 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.204923 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.204929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:48.205184 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:48.206127 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:48.206170 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:48.209298 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:48.214038 (+ 4740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:48.214050 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:48.214079 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:48.214764 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:48.215296 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:48.215302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:48.215306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:48.215680 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:48.216400 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:48.216410 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:48.216898 (+ 488us) client_negotiation.cc:769] Sending connection context
0927 22:01:48.217240 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 22:01:48.217978 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":449,"threads_started":1}
I20260927 22:01:48.221410 9825 heartbeater.cc:347] Connected to a master server at 127.3.159.62:34529
I20260927 22:01:48.221836 9825 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:48.222836 9825 heartbeater.cc:511] Master 127.3.159.62:34529 requested a full tablet report, sending...
I20260927 22:01:48.224922 9368 ts_manager.cc:194] Registered new tserver with Master: e3f9a255a7c3475f8c70ccbadb8d2af9 (127.3.159.3:32947)
I20260927 22:01:48.225924 9368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:56457
I20260927 22:01:48.228941 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/test-admin.keytab.
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:48.261839 3708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 22:01:48.270249 9833 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:01:48.265594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.3.159.1:45451)
0927 22:01:48.266523 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:48.266536 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:48.266989 (+ 453us) server_negotiation.cc:407] Connection header received
0927 22:01:48.267233 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:48.267242 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:48.267543 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:48.267741 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:48.269647 (+ 1906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37676: BlockingRecv error: recv got EOF from 127.0.0.1:37676 (error 108)
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":441,"threads_started":1}
W20260927 22:01:48.270443 3708 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5cc)
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:48.272787 3708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:48.275220 3708 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:01:48.299839 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:48.277657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37684 (local address 127.3.159.1:45451)
0927 22:01:48.277990 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:48.277998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:48.278363 (+ 365us) server_negotiation.cc:407] Connection header received
0927 22:01:48.278450 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:48.278455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:48.278540 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:48.278657 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:48.280221 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.281890 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.283619 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.284458 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.284621 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:48.291128 (+ 6507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:48.291153 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:48.291177 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:48.291276 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:48.295348 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.296551 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.296561 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.296564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.296803 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.297654 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.297661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.297664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.298297 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:48.298652 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:48.299053 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 22:01:48.299510 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260927 22:01:48.309055 9833 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:01:48.306804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.3.159.1:45451)
0927 22:01:48.307021 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:48.307026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:48.307049 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:01:48.307251 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:48.307257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:48.307367 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:48.307499 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:48.308586 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37696: BlockingRecv error: recv got EOF from 127.0.0.1:37696 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 22:01:48.309587 3708 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5cc)
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:48.311888 3708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:48.313797 3708 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:01:48.333595 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:48.315674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.3.159.1:45451)
0927 22:01:48.316076 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:48.316084 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:48.316432 (+ 348us) server_negotiation.cc:407] Connection header received
0927 22:01:48.316601 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:48.316606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:48.316690 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:48.316810 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:48.318244 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.319048 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.320672 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.321200 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.321356 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:48.326711 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:48.326718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:48.326722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:48.326793 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:48.329743 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.330907 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.330913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.330915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.331103 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.331744 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.331749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.331752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.332099 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:48.332328 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:48.332864 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 22:01:48.333141 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
W20260927 22:01:48.339428 9833 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:01:48.337030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.3.159.1:45451)
0927 22:01:48.337547 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:48.337555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:48.337861 (+ 306us) server_negotiation.cc:407] Connection header received
0927 22:01:48.338114 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:48.338119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:48.338209 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:48.338320 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:48.339140 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37714: BlockingRecv error: recv got EOF from 127.0.0.1:37714 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
W20260927 22:01:48.339875 3708 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790546447244210-3708-0/krb5kdc/krb5cc' of type FILE
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:48.341653 3708 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:48.343044 3708 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:01:48.363875 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:48.344917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.3.159.1:45451)
0927 22:01:48.345160 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:48.345167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:48.345706 (+ 539us) server_negotiation.cc:407] Connection header received
0927 22:01:48.345938 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:48.345942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:48.346005 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:48.346115 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:48.347194 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.347938 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.349235 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.349753 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.349896 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:48.354532 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:48.354539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:48.354544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:48.354604 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:48.358912 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.360811 (+ 1899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.360818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.360821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.361144 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.361891 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.361895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.361898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.362293 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:48.362620 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:48.363070 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 22:01:48.363444 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:01:48.366612 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 9429
I20260927 22:01:48.385016 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:48.367939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.3.159.1:45451)
0927 22:01:48.368166 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:48.368171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:48.368605 (+ 434us) server_negotiation.cc:407] Connection header received
0927 22:01:48.368959 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:48.368963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:48.369046 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:48.369156 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:48.370568 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.371491 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.373524 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.374057 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.374211 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:48.376795 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:48.376802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:48.376806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:48.376871 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:48.380638 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.382197 (+ 1559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.382202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.382205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.382418 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.382982 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.382987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.382989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.383383 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:48.383713 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:48.384394 (+ 681us) server_negotiation.cc:299] Negotiation successful
0927 22:01:48.384690 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 22:01:48.386022 9552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:48.482527 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 9429
I20260927 22:01:48.501950 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 9563
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:01:48.523283 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:48.503226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.3.159.2:43959)
0927 22:01:48.503975 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:48.503984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:48.504017 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:01:48.504106 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:48.504114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:48.504401 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:48.504635 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:48.506007 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.507351 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.508950 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.509831 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.510007 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:48.515139 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:48.515160 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:48.515182 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:48.515238 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:48.519206 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.520350 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.520361 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.520364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.520546 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.521178 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.521184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.521188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.521728 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:48.522152 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:48.522505 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 22:01:48.522853 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":358,"threads_started":1}
I20260927 22:01:48.525089 9686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:48.619014 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 9563
I20260927 22:01:48.637279 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 9697
Sep 27 22:01:48 dist-test-slave-sg4j krb5kdc[9323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:01:48.657975 9848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:48.638778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.3.159.3:32947)
0927 22:01:48.639563 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:48.639572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:48.639614 (+ 42us) server_negotiation.cc:407] Connection header received
0927 22:01:48.639697 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:48.639705 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:48.639999 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:48.640198 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:48.641672 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.642939 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.644600 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:48.645270 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:48.645557 (+ 287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:48.650431 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:48.650449 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:48.650467 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:48.650519 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:48.654024 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.655317 (+ 1293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.655330 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.655332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.655525 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:48.656039 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:48.656045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:48.656048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:48.656540 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:48.656846 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:48.657272 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:01:48.657627 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":448,"threads_started":1}
I20260927 22:01:48.659538 9820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:48.754853 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 9697
I20260927 22:01:48.771731 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 9336
I20260927 22:01:48.773358 9398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:48.894862 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 9336
2026-09-27T22:01:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4562 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:01:49 dist-test-slave-sg4j krb5kdc[9853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:01:49 dist-test-slave-sg4j krb5kdc[9853](info): set up 2 sockets
Sep 27 22:01:49 dist-test-slave-sg4j krb5kdc[9853](info): commencing operation
krb5kdc: starting...
W20260927 22:01:50.034785 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:01:50 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:01:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:01:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.62.keytab.
Entry for principal oryx/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.62.keytab.
I20260927 22:01:50.307049 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:33985
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.62.keytab
--principal=oryx/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:33985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:50.718724 9866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:50.719127 9866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:50.719383 9866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:50.727804 9866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:50.727964 9866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:50.728042 9866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:50.728142 9866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:50.728248 9866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:01:50.739941 9866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:33985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.62.keytab
--principal=oryx/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:33985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:50.741869 9866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:50.743645 9866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:50.759446 9874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:50.759550 9871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:50.759550 9872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:50.761092 9866 server_base.cc:1034] running on GCE node
I20260927 22:01:50.762387 9866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:50.764485 9866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:50.767925 9866 hybrid_clock.cc:648] HybridClock initialized: now 1790546510767723 us; error 164 us; skew 500 ppm
I20260927 22:01:50.768505 9866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:50 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546510, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:50.779520 9866 init.cc:377] Logged in from keytab as oryx/127.3.159.62@KRBTEST.COM (short username oryx)
I20260927 22:01:50.782021 9866 webserver.cc:460] Webserver started at http://127.3.159.62:44881/ using document root <none> and password file <none>
I20260927 22:01:50.783035 9866 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:50.783242 9866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:50.783638 9866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:50.787477 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "082117a1043840558566f1322649c58b"
format_stamp: "Formatted at 2026-09-27 22:01:50 on dist-test-slave-sg4j"
server_key: "3d8b119c84da66296a713ddc7c2a5019"
server_key_iv: "e3c86ec34b701ad1ddcc218ff0712724"
server_key_version: "encryptionkey@0"
I20260927 22:01:50.788390 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "082117a1043840558566f1322649c58b"
format_stamp: "Formatted at 2026-09-27 22:01:50 on dist-test-slave-sg4j"
server_key: "3d8b119c84da66296a713ddc7c2a5019"
server_key_iv: "e3c86ec34b701ad1ddcc218ff0712724"
server_key_version: "encryptionkey@0"
I20260927 22:01:50.795392 9866 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 22:01:50.800366 9881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:50.802541 9866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:01:50.802840 9866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "082117a1043840558566f1322649c58b"
format_stamp: "Formatted at 2026-09-27 22:01:50 on dist-test-slave-sg4j"
server_key: "3d8b119c84da66296a713ddc7c2a5019"
server_key_iv: "e3c86ec34b701ad1ddcc218ff0712724"
server_key_version: "encryptionkey@0"
I20260927 22:01:50.803103 9866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:50.816397 9866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:50.821806 9866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:50.822180 9866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:50.846446 9866 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:33985
I20260927 22:01:50.846493 9932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:33985 every 8 connection(s)
I20260927 22:01:50.848301 9866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:01:50.852881 9933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:50.855628 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 9866
I20260927 22:01:50.855952 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:01:50.856734 3708 external_mini_cluster.cc:1428] Setting key 17a13bb6aef04c03405b17f656007a33
I20260927 22:01:50.868227 9933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b: Bootstrap starting.
I20260927 22:01:50.872469 9933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:50.874006 9933 log.cc:824] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:01:50 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:01:50.878131 9933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b: No bootstrap required, opened a new log
I20260927 22:01:50.890535 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:50.860323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.3.159.62:33985)
0927 22:01:50.861339 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:50.861354 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:50.861458 (+ 104us) server_negotiation.cc:407] Connection header received
0927 22:01:50.863425 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:50.863457 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:50.864048 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:50.864513 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:50.866628 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:50.868476 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:50.870572 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:50.871596 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:50.871806 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:50.877765 (+ 5959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:50.877777 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:50.877800 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:50.877854 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:50.882935 (+ 5081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:50.884076 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:50.884089 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:50.884120 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:50.884389 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:50.884900 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:50.884905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:50.884909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:50.885801 (+ 892us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:50.886147 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:50.886806 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 22:01:50.887504 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":454,"threads_started":1}
I20260927 22:01:50.891602 9933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } }
I20260927 22:01:50.892102 9933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:50.892223 9933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082117a1043840558566f1322649c58b, State: Initialized, Role: FOLLOWER
I20260927 22:01:50.893124 9933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } }
I20260927 22:01:50.893569 9933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:01:50.893752 9933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:01:50.893970 9933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:50.897545 9933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } }
I20260927 22:01:50.898288 9933 leader_election.cc:302] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 082117a1043840558566f1322649c58b; no voters:
I20260927 22:01:50.899027 9933 leader_election.cc:288] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:01:50.899276 9938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:50.900465 9938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 1 LEADER]: Becoming Leader. State: Replica: 082117a1043840558566f1322649c58b, State: Running, Role: LEADER
I20260927 22:01:50.901208 9938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } }
I20260927 22:01:50.902552 9933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:50.910794 9939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } }
I20260927 22:01:50.911408 9939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:50.912295 9940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 082117a1043840558566f1322649c58b. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } }
I20260927 22:01:50.912832 9940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:50.926004 9948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:50.934296 9948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:50.950444 9948 catalog_manager.cc:1347] Generated new cluster ID: 783c9c685ebe49c9a2ad6864a3e32582
I20260927 22:01:50.950600 9948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:50.983975 9948 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:01:50.985836 9948 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:50.995215 9948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b: Generated new TSK 0
I20260927 22:01:50.996446 9948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab.
Entry for principal oryx/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab.
I20260927 22:01:51.074849 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:33985
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab
--principal=oryx/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:51.552368 9960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:51.552754 9960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:51.552940 9960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:51.561369 9960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:51.561548 9960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:51.561800 9960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:01:51.573158 9960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab
--principal=oryx/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:33985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:51.574985 9960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:51.576735 9960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:51.594514 9966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:51.594622 9965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:51.595482 9968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:51.599834 9960 server_base.cc:1034] running on GCE node
I20260927 22:01:51.600778 9960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:51.602257 9960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:51.605682 9960 hybrid_clock.cc:648] HybridClock initialized: now 1790546511605551 us; error 104 us; skew 500 ppm
I20260927 22:01:51.606207 9960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:51 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546511, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:51.611001 9960 init.cc:377] Logged in from keytab as oryx/127.3.159.1@KRBTEST.COM (short username oryx)
I20260927 22:01:51.613499 9960 webserver.cc:460] Webserver started at http://127.3.159.1:32969/ using document root <none> and password file <none>
I20260927 22:01:51.614563 9960 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:51.614751 9960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:51.615154 9960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:51.619158 9960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "00d4d325d1014c8d9daeee0118430882"
format_stamp: "Formatted at 2026-09-27 22:01:51 on dist-test-slave-sg4j"
server_key: "dc00ff2a92bde921527f737d5aa908fb"
server_key_iv: "a295675d0bcf9319c09701cd8fb60964"
server_key_version: "encryptionkey@0"
I20260927 22:01:51.620204 9960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "00d4d325d1014c8d9daeee0118430882"
format_stamp: "Formatted at 2026-09-27 22:01:51 on dist-test-slave-sg4j"
server_key: "dc00ff2a92bde921527f737d5aa908fb"
server_key_iv: "a295675d0bcf9319c09701cd8fb60964"
server_key_version: "encryptionkey@0"
I20260927 22:01:51.627621 9960 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260927 22:01:51.633133 9975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:51.635731 9960 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260927 22:01:51.636073 9960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "00d4d325d1014c8d9daeee0118430882"
format_stamp: "Formatted at 2026-09-27 22:01:51 on dist-test-slave-sg4j"
server_key: "dc00ff2a92bde921527f737d5aa908fb"
server_key_iv: "a295675d0bcf9319c09701cd8fb60964"
server_key_version: "encryptionkey@0"
I20260927 22:01:51.636417 9960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:51.679200 9960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:51.685138 9960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:51.685612 9960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:51.687054 9960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:51.689139 9960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:51.689289 9960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:51.689513 9960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:51.689603 9960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:51.736788 9960 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:37633
I20260927 22:01:51.736914 10087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:37633 every 8 connection(s)
I20260927 22:01:51.738667 9960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:01:51.741853 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 9960
I20260927 22:01:51.742113 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:01:51.742983 3708 external_mini_cluster.cc:1428] Setting key f62ad500b897c30b78555957708322d1
Sep 27 22:01:51 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546511, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:01:51.767204 10090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:51.742281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:60303 (local address 127.3.159.62:33985)
0927 22:01:51.742942 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:51.742950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:51.744494 (+ 1544us) server_negotiation.cc:407] Connection header received
0927 22:01:51.746881 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:51.746891 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:51.746999 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:51.747150 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:51.749360 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:51.750254 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:51.752664 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:51.753138 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:51.753427 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:51.758119 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:51.758129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:51.758136 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:51.758208 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:51.762286 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:51.763701 (+ 1415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:51.763710 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:51.763717 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:51.763914 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:51.764520 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:51.764529 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:51.764535 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:51.764985 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:51.765173 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:51.766349 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0927 22:01:51.766691 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":317,"threads_started":1}
WARNING: no policy specified for oryx/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:01:51.768244 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:51.743217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.1:60303)
0927 22:01:51.744247 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:51.744318 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:51.746495 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:51.747484 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:51.747498 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:51.748258 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:51.749060 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:51.749084 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:51.750455 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:51.750460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:51.752361 (+ 1901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:51.752386 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:51.753571 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:51.753577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:51.753848 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:51.754722 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:51.754757 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:51.757751 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:51.762534 (+ 4783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:51.762580 (+ 46us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:51.762608 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:51.763506 (+ 898us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:51.764103 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:51.764106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:51.764109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:51.764351 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:51.765160 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:51.765168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:51.765759 (+ 591us) client_negotiation.cc:769] Sending connection context
0927 22:01:51.766078 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 22:01:51.766664 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":444,"threads_started":1}
I20260927 22:01:51.770677 10088 heartbeater.cc:347] Connected to a master server at 127.3.159.62:33985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.2@KRBTEST.COM" created.
I20260927 22:01:51.771128 10088 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:51.771992 10088 heartbeater.cc:511] Master 127.3.159.62:33985 requested a full tablet report, sending...
I20260927 22:01:51.775002 9898 ts_manager.cc:194] Registered new tserver with Master: 00d4d325d1014c8d9daeee0118430882 (127.3.159.1:37633)
I20260927 22:01:51.777542 9898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:60303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab.
Entry for principal oryx/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab.
I20260927 22:01:51.807215 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:33985
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab
--principal=oryx/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:01:52.204228 10094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:52.204623 10094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:52.204808 10094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:52.213094 10094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:52.213306 10094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:52.213541 10094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:01:52.226007 10094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab
--principal=oryx/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:33985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:52.227535 10094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:52.228957 10094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:52.245052 10100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:52.245157 10099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:52.245427 10102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:52.251770 10094 server_base.cc:1034] running on GCE node
I20260927 22:01:52.252668 10094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:52.254102 10094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:52.255406 10094 hybrid_clock.cc:648] HybridClock initialized: now 1790546512255338 us; error 57 us; skew 500 ppm
I20260927 22:01:52.255860 10094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:52 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546512, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:52.260198 10094 init.cc:377] Logged in from keytab as oryx/127.3.159.2@KRBTEST.COM (short username oryx)
I20260927 22:01:52.262641 10094 webserver.cc:460] Webserver started at http://127.3.159.2:46185/ using document root <none> and password file <none>
I20260927 22:01:52.263625 10094 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:52.263758 10094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:52.264055 10094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:52.267969 10094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "11ff1411e4e14671a218145ee48c2962"
format_stamp: "Formatted at 2026-09-27 22:01:52 on dist-test-slave-sg4j"
server_key: "d38563af38b468b75845cf680e56f7cf"
server_key_iv: "0f3576356ae393b4ab2aa74f864bc664"
server_key_version: "encryptionkey@0"
I20260927 22:01:52.268836 10094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "11ff1411e4e14671a218145ee48c2962"
format_stamp: "Formatted at 2026-09-27 22:01:52 on dist-test-slave-sg4j"
server_key: "d38563af38b468b75845cf680e56f7cf"
server_key_iv: "0f3576356ae393b4ab2aa74f864bc664"
server_key_version: "encryptionkey@0"
I20260927 22:01:52.276469 10094 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 22:01:52.281620 10109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:52.283777 10094 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:01:52.284068 10094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "11ff1411e4e14671a218145ee48c2962"
format_stamp: "Formatted at 2026-09-27 22:01:52 on dist-test-slave-sg4j"
server_key: "d38563af38b468b75845cf680e56f7cf"
server_key_iv: "0f3576356ae393b4ab2aa74f864bc664"
server_key_version: "encryptionkey@0"
I20260927 22:01:52.284375 10094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:52.300992 10094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:52.306951 10094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:52.307375 10094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:52.308794 10094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:52.310611 10094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:52.310729 10094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:52.310923 10094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:52.311012 10094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:52.353317 10094 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:33831
I20260927 22:01:52.353451 10221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:33831 every 8 connection(s)
I20260927 22:01:52.355041 10094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:01:52.356745 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 10094
I20260927 22:01:52.357007 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:01:52.357571 3708 external_mini_cluster.cc:1428] Setting key f9af4985129e429d726fe542247cdde5
Sep 27 22:01:52 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546512, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.159.3@KRBTEST.COM; defaulting to no policy
I20260927 22:01:52.383556 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:52.358893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:51983 (local address 127.3.159.62:33985)
0927 22:01:52.359572 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:52.359577 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:52.361014 (+ 1437us) server_negotiation.cc:407] Connection header received
0927 22:01:52.363131 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:52.363153 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:52.363274 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:52.363450 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:52.365622 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:52.366929 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:52.370425 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:52.370962 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:52.371183 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:52.376595 (+ 5412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:52.376602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:52.376605 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:52.376652 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:52.379853 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:52.380862 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:52.380867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:52.380870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:52.380996 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:52.381522 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:52.381531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:52.381534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:52.381851 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:52.382025 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:52.382890 (+ 865us) server_negotiation.cc:299] Negotiation successful
0927 22:01:52.383155 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":1160,"threads_started":1}
I20260927 22:01:52.384948 10225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:52.359824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.2:51983)
0927 22:01:52.360795 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:52.360848 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:52.362782 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:52.363843 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:52.363853 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:52.364494 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:52.365303 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:52.365326 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:52.367242 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:52.367248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:52.370098 (+ 2850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:52.370135 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:52.371301 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:52.371313 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:52.371760 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:52.372754 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:52.372796 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:52.376222 (+ 3426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:52.380070 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:52.380082 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:52.380114 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:52.380703 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:52.381114 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:52.381119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:52.381123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:52.381359 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:52.381985 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:52.381996 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:52.382462 (+ 466us) client_negotiation.cc:769] Sending connection context
0927 22:01:52.382781 (+ 319us) client_negotiation.cc:240] Negotiation successful
0927 22:01:52.383459 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":406,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.3@KRBTEST.COM" created.
I20260927 22:01:52.387126 10222 heartbeater.cc:347] Connected to a master server at 127.3.159.62:33985
I20260927 22:01:52.387634 10222 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:52.388561 10222 heartbeater.cc:511] Master 127.3.159.62:33985 requested a full tablet report, sending...
I20260927 22:01:52.391201 9898 ts_manager.cc:194] Registered new tserver with Master: 11ff1411e4e14671a218145ee48c2962 (127.3.159.2:33831)
I20260927 22:01:52.392180 9898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:51983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab.
Entry for principal oryx/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab.
I20260927 22:01:52.420603 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:33985
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab
--principal=oryx/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:52.782328 10088 heartbeater.cc:503] Master 127.3.159.62:33985 was elected leader, sending a full tablet report...
W20260927 22:01:52.811568 10228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:52.812000 10228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:52.812211 10228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:52.820791 10228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:52.820981 10228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:52.821180 10228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:01:52.832556 10228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab
--principal=oryx/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:33985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:52.834069 10228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:52.835402 10228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:52.851219 10234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:52.851588 10236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:52.851981 10233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:52.853004 10228 server_base.cc:1034] running on GCE node
I20260927 22:01:52.853861 10228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:52.855449 10228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:52.856894 10228 hybrid_clock.cc:648] HybridClock initialized: now 1790546512856794 us; error 88 us; skew 500 ppm
I20260927 22:01:52.857335 10228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:52 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546512, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:52.861126 10228 init.cc:377] Logged in from keytab as oryx/127.3.159.3@KRBTEST.COM (short username oryx)
I20260927 22:01:52.863026 10228 webserver.cc:460] Webserver started at http://127.3.159.3:44529/ using document root <none> and password file <none>
I20260927 22:01:52.863850 10228 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:52.864004 10228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:52.864363 10228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:52.867663 10228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "60b8c7e9f16e49fb9a6538dec710976a"
format_stamp: "Formatted at 2026-09-27 22:01:52 on dist-test-slave-sg4j"
server_key: "42d7c33517d97e4bda4536be2f2525f7"
server_key_iv: "fdede79bc7ffe63c6a4c98b2101600d8"
server_key_version: "encryptionkey@0"
I20260927 22:01:52.868525 10228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "60b8c7e9f16e49fb9a6538dec710976a"
format_stamp: "Formatted at 2026-09-27 22:01:52 on dist-test-slave-sg4j"
server_key: "42d7c33517d97e4bda4536be2f2525f7"
server_key_iv: "fdede79bc7ffe63c6a4c98b2101600d8"
server_key_version: "encryptionkey@0"
I20260927 22:01:52.875120 10228 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 22:01:52.879685 10243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:52.881852 10228 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 22:01:52.882098 10228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "60b8c7e9f16e49fb9a6538dec710976a"
format_stamp: "Formatted at 2026-09-27 22:01:52 on dist-test-slave-sg4j"
server_key: "42d7c33517d97e4bda4536be2f2525f7"
server_key_iv: "fdede79bc7ffe63c6a4c98b2101600d8"
server_key_version: "encryptionkey@0"
I20260927 22:01:52.882318 10228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:52.898387 10228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:52.904364 10228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:52.904964 10228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:52.906899 10228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:52.909271 10228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:52.909469 10228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:52.909626 10228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:52.909690 10228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:52.960536 10228 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:34903
I20260927 22:01:52.960716 10355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:34903 every 8 connection(s)
I20260927 22:01:52.962216 10228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:01:52.969627 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 10228
I20260927 22:01:52.969913 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:01:52.970551 3708 external_mini_cluster.cc:1428] Setting key 68fde91f3df35461f06f1c94050f0fdd
Sep 27 22:01:52 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546512, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:01:52.989176 10358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:52.965788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:59213 (local address 127.3.159.62:33985)
0927 22:01:52.966352 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:52.966359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:52.967745 (+ 1386us) server_negotiation.cc:407] Connection header received
0927 22:01:52.970668 (+ 2923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:52.970680 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:52.970806 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:52.970967 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:52.973688 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:52.974764 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:52.976734 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:52.977158 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:52.977278 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:52.981672 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:52.981678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:52.981683 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:52.981725 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:52.984594 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:52.985923 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:52.985932 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:52.985936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:52.986126 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:52.986780 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:52.986786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:52.986789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:52.987272 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:52.987463 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:52.988565 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0927 22:01:52.988764 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":265,"threads_started":1}
I20260927 22:01:52.990480 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:52.966359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.3:59213)
0927 22:01:52.967478 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:52.967536 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:52.970151 (+ 2615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:52.971370 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:52.971382 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:52.972259 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:52.973458 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:52.973497 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:52.974936 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:52.974941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:52.976542 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:52.976555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:52.977472 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:52.977480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:52.977746 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:52.978555 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:52.978581 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:52.981205 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:52.984804 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:52.984816 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:52.984838 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:52.985654 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:52.986273 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:52.986282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:52.986286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:52.986574 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:52.987479 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:52.987492 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:52.988084 (+ 592us) client_negotiation.cc:769] Sending connection context
0927 22:01:52.988395 (+ 311us) client_negotiation.cc:240] Negotiation successful
0927 22:01:52.988940 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":441,"threads_started":1}
I20260927 22:01:52.992188 10356 heartbeater.cc:347] Connected to a master server at 127.3.159.62:33985
I20260927 22:01:52.992619 10356 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:52.993525 10356 heartbeater.cc:511] Master 127.3.159.62:33985 requested a full tablet report, sending...
I20260927 22:01:52.995225 9898 ts_manager.cc:194] Registered new tserver with Master: 60b8c7e9f16e49fb9a6538dec710976a (127.3.159.3:34903)
I20260927 22:01:52.996218 9898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:59213
I20260927 22:01:53.001367 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.159.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.61.keytab.
Entry for principal oryx/127.3.159.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.61.keytab.
I20260927 22:01:53.062043 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.61:34347
--webserver_interface=127.3.159.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.61.keytab
--principal=oryx/127.3.159.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:33985,127.3.159.61:34347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:53.395553 10222 heartbeater.cc:503] Master 127.3.159.62:33985 was elected leader, sending a full tablet report...
W20260927 22:01:53.477583 10362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:53.478034 10362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:53.478215 10362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:53.486661 10362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:53.486876 10362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:53.486974 10362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:53.487104 10362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:53.487195 10362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:01:53.499226 10362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:33985,127.3.159.61:34347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.61.keytab
--principal=oryx/127.3.159.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.61:34347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.159.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
W20260927 22:01:53.499787 10362 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.159.62:33985,127.3.159.61:34347'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 22:01:53.501243 10362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:53.502959 10362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:53.518841 10367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:53.519032 10368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:53.519060 10370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:53.520412 10362 server_base.cc:1034] running on GCE node
I20260927 22:01:53.521731 10362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:53.523548 10362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:53.524883 10362 hybrid_clock.cc:648] HybridClock initialized: now 1790546513524790 us; error 66 us; skew 500 ppm
I20260927 22:01:53.525348 10362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:53 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546513, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:53.529286 10362 init.cc:377] Logged in from keytab as oryx/127.3.159.61@KRBTEST.COM (short username oryx)
I20260927 22:01:53.531138 10362 webserver.cc:460] Webserver started at http://127.3.159.61:45825/ using document root <none> and password file <none>
I20260927 22:01:53.532017 10362 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:53.532152 10362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:53.532501 10362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:53.535812 10362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data/instance:
uuid: "9ffa375e09fa444187a77ac58b5966ee"
format_stamp: "Formatted at 2026-09-27 22:01:53 on dist-test-slave-sg4j"
server_key: "d8fb519fa818adafa732cda3bbce4c55"
server_key_iv: "8766fea4c1a55730f9d76c02c069fa4e"
server_key_version: "encryptionkey@0"
I20260927 22:01:53.536702 10362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/wal/instance:
uuid: "9ffa375e09fa444187a77ac58b5966ee"
format_stamp: "Formatted at 2026-09-27 22:01:53 on dist-test-slave-sg4j"
server_key: "d8fb519fa818adafa732cda3bbce4c55"
server_key_iv: "8766fea4c1a55730f9d76c02c069fa4e"
server_key_version: "encryptionkey@0"
I20260927 22:01:53.543262 10362 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260927 22:01:53.548375 10377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:53.550901 10362 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:01:53.551179 10362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/wal
uuid: "9ffa375e09fa444187a77ac58b5966ee"
format_stamp: "Formatted at 2026-09-27 22:01:53 on dist-test-slave-sg4j"
server_key: "d8fb519fa818adafa732cda3bbce4c55"
server_key_iv: "8766fea4c1a55730f9d76c02c069fa4e"
server_key_version: "encryptionkey@0"
I20260927 22:01:53.551443 10362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:53.566334 10362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:53.571702 10362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:53.572023 10362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:53.596493 10362 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.61:34347
I20260927 22:01:53.596537 10428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.61:34347 every 8 connection(s)
I20260927 22:01:53.598152 10362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
I20260927 22:01:53.603171 10429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:53.603533 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 10362
I20260927 22:01:53.615686 10429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:01:53 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.159.61@KRBTEST.COM
Sep 27 22:01:53 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546513, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.61@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:01:53.633229 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.606327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.3.159.61:34347)
0927 22:01:53.607488 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:53.607507 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:53.607572 (+ 65us) server_negotiation.cc:407] Connection header received
0927 22:01:53.610697 (+ 3125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:53.610730 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:53.611293 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:53.611859 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:53.613460 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.614880 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.616212 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.617093 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.617317 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.623067 (+ 5750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:53.623077 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:53.623091 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:53.623159 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:53.626784 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.628055 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.628063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.628070 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.628302 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.628919 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.628925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.628929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.629940 (+ 1011us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:53.630219 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:53.630840 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 22:01:53.631360 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":494,"threads_started":1}
I20260927 22:01:53.639258 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.619326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46608)
0927 22:01:53.620368 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:53.620421 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:53.620676 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:53.621164 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:53.621172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:53.622111 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:53.622453 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.622464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.623707 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.623716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.625111 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.625125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.626139 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.626146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.626369 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.627216 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:53.627245 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:53.630733 (+ 3488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:53.635612 (+ 4879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.635627 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.635634 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.636181 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.637118 (+ 937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.637122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.637125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.637269 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.638113 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:53.638121 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:53.638358 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 22:01:53.638565 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 22:01:53.638902 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":594,"threads_started":1}
I20260927 22:01:53.639356 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.619273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.3.159.62:33985)
0927 22:01:53.619937 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:53.619947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:53.620555 (+ 608us) server_negotiation.cc:407] Connection header received
0927 22:01:53.620833 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:53.620840 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:53.620941 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:53.621082 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:53.622626 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.623533 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.625337 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.625890 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.626071 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.630995 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:53.631003 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:53.631007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:53.631073 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:53.635430 (+ 4357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.636351 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.636358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.636365 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.636522 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.637527 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.637532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.637534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.637954 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:53.638103 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:53.638794 (+ 691us) server_negotiation.cc:299] Negotiation successful
0927 22:01:53.639062 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":347,"threads_started":1}
I20260927 22:01:53.641796 10429 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:01:53 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546513, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.61@KRBTEST.COM for oryx/127.3.159.61@KRBTEST.COM
I20260927 22:01:53.657862 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.642653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:34347 (local address 127.0.0.1:32924)
0927 22:01:53.642927 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:53.642959 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:53.643102 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:53.643502 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:53.643507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:53.643962 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:53.644130 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.644139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.645267 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.645273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.646404 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.646416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.647213 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.647220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.647447 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.648267 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:53.648343 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:53.650874 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:53.654504 (+ 3630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.654511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.654516 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.654952 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.655647 (+ 695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.655651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.655653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.655798 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.656820 (+ 1022us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:53.656824 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:53.657012 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 22:01:53.657154 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:01:53.657561 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 22:01:53.658219 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.643000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.3.159.61:34347)
0927 22:01:53.643234 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:53.643240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:53.643257 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:01:53.643310 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:53.643314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:53.643375 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:53.643489 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:53.644281 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.645108 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.646557 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.647031 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.647192 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.651121 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:53.651127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:53.651130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:53.651161 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:53.654306 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.655315 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.655321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.655325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.655493 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.656026 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.656034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.656037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.656645 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:53.656836 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:53.657288 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 22:01:53.657770 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":75}
I20260927 22:01:53.665292 10429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: Bootstrap starting.
I20260927 22:01:53.669845 10429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:53.671350 10429 log.cc:824] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:53.675573 10429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: No bootstrap required, opened a new log
I20260927 22:01:53.684401 10429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } }
I20260927 22:01:53.684903 10429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:53.685073 10429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ffa375e09fa444187a77ac58b5966ee, State: Initialized, Role: FOLLOWER
I20260927 22:01:53.685917 10429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } }
I20260927 22:01:53.687279 10436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } } }
I20260927 22:01:53.687798 10436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:01:53.688838 10429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:01:53.700277 10447 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:01:53.700469 10447 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:01:53.704295 10362 master_runner.cc:428] Detected that this master 9ffa375e09fa444187a77ac58b5966ee is joining an existing cluster
I20260927 22:01:53.704425 10362 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.159.61:34347
I20260927 22:01:53.728297 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.712017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:34347 (local address 127.0.0.1:32930)
0927 22:01:53.712847 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:53.712861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:53.713051 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:53.713424 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:53.713429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:53.714113 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:53.714543 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.714554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.715673 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.715678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.716848 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.716860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.717792 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.717798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.718112 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.718902 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:53.718923 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:53.719988 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:53.724722 (+ 4734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.724729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.724735 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.725448 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.726095 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.726101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.726105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.726301 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.727016 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:53.727021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:53.727248 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 22:01:53.727408 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 22:01:53.727761 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":103,"spinlock_wait_cycles":73472,"thread_start_us":471,"threads_started":1}
I20260927 22:01:53.729272 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.712455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.3.159.61:34347)
0927 22:01:53.712635 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:53.712641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:53.712953 (+ 312us) server_negotiation.cc:407] Connection header received
0927 22:01:53.713178 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:53.713183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:53.713254 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:53.713373 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:53.714678 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.715490 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.717008 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.717592 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.717739 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.720395 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:53.720407 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:53.720412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:53.720442 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:53.724322 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.725674 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.725680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.725684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.725905 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.726455 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.726459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.726462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.726861 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:53.727064 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:53.728666 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0927 22:01:53.728885 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":31616}
I20260927 22:01:53.729984 10453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.710537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46616)
0927 22:01:53.711681 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:53.711715 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:53.711929 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:53.712656 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:53.712663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:53.713822 (+ 1159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:53.714308 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.714336 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.716073 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.716078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.718042 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.718060 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.719567 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.719574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.719804 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.720481 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:53.720508 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:53.722043 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:53.726594 (+ 4551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.726600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.726604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.727130 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.727599 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.727604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.727608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.727806 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.728758 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:53.728767 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:53.728973 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 22:01:53.729139 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 22:01:53.729528 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":6,"thread_start_us":713,"threads_started":1}
I20260927 22:01:53.730707 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.710483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.3.159.62:33985)
0927 22:01:53.710757 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:53.710781 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:53.711854 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 22:01:53.712106 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:53.712114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:53.712244 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:53.712428 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:53.714583 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.715795 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.718264 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.719022 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.719153 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.722340 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:53.722346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:53.722350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:53.722389 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:53.726373 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.727298 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.727303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.727307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.727471 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.727968 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.727973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.727977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.728357 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:53.728694 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:53.729691 (+ 997us) server_negotiation.cc:299] Negotiation successful
0927 22:01:53.730289 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
Sep 27 22:01:53 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546510, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.62@KRBTEST.COM for oryx/127.3.159.61@KRBTEST.COM
I20260927 22:01:53.758616 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.737459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:34347 (local address 127.0.0.1:32944)
0927 22:01:53.738227 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:53.738273 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:53.738529 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:53.739262 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:53.739269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:53.740244 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:53.740541 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.740553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.742482 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.742491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.744371 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.744392 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.745730 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.745740 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.746022 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.746971 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:53.747006 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:53.749905 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:53.754757 (+ 4852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.754777 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.754789 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.755439 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.755935 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.755942 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.755948 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.756247 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.757131 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:53.757143 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:53.757364 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 22:01:53.757607 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 22:01:53.758034 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":499,"threads_started":1}
I20260927 22:01:53.760382 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.737749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.3.159.61:34347)
0927 22:01:53.737948 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:53.737955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:53.738373 (+ 418us) server_negotiation.cc:407] Connection header received
0927 22:01:53.738799 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:53.738803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:53.738899 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:53.738959 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:53.740828 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.742122 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.744618 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.745474 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.745660 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.750158 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:53.750164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:53.750167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:53.750203 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:53.754535 (+ 4332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.755614 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.755623 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.755630 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.755799 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.756393 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.756400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.756407 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.756899 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:53.757046 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:53.759148 (+ 2102us) server_negotiation.cc:299] Negotiation successful
0927 22:01:53.759866 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:01:53.763542 9898 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.159.61:34347: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee"
member_type: NON_VOTER
last_known_addr {
host: "127.3.159.61"
port: 34347
}
attrs {
promote: true
}
}
dest_uuid: "082117a1043840558566f1322649c58b"
cas_config_opid_index: -1
I20260927 22:01:53.764866 9898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.3.159.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.60@KRBTEST.COM" created.
I20260927 22:01:53.768594 10459 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9ffa375e09fa444187a77ac58b5966ee (127.3.159.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } } }
I20260927 22:01:53.770855 10404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:53.773852 10459 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.159.61:34347
I20260927 22:01:53.774417 10459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } } }
I20260927 22:01:53.774662 10459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:53.774998 10362 master.cc:537] Master@127.3.159.61:34347 shutting down...
I20260927 22:01:53.779525 10404 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 FOLLOWER]: Refusing update from remote peer 082117a1043840558566f1322649c58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 22:01:53.792717 10362 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:53.793251 10362 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:53.793514 10362 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: stopping tablet replica
I20260927 22:01:53.793650 10459 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:01:53.797636 9882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from unknown peer (error 108)
I20260927 22:01:53.799431 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.798943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:34347 (local address 127.0.0.1:32950)
0927 22:01:53.799128 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:53.799239 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:34347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 22:01:53.799825 9882 consensus_peers.cc:487] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b -> Peer 9ffa375e09fa444187a77ac58b5966ee (127.3.159.61:34347): Couldn't send request to peer 9ffa375e09fa444187a77ac58b5966ee. Status: Network error: Client connection negotiation failed: client connection to 127.3.159.61:34347: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:01:53.802357 10362 master.cc:552] Master@127.3.159.61:34347 shutdown complete.
I20260927 22:01:53.802474 10362 master_runner.cc:305] Clearing existing system tablet
I20260927 22:01:53.802922 10362 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:53.803063 10362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:53.807760 10362 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:01:53.809671 10464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:53.810266 10362 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:53.810617 10362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:01:53.810720 10362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/wal
uuid: "9ffa375e09fa444187a77ac58b5966ee"
format_stamp: "Formatted at 2026-09-27 22:01:53 on dist-test-slave-sg4j"
server_key: "d8fb519fa818adafa732cda3bbce4c55"
server_key_iv: "8766fea4c1a55730f9d76c02c069fa4e"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.60.keytab.
Entry for principal oryx/127.3.159.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.60.keytab.
I20260927 22:01:53.815117 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.60:34919
--webserver_interface=127.3.159.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.60.keytab
--principal=oryx/127.3.159.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:33985,127.3.159.61:34347,127.3.159.60:34919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:53.816862 10362 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:53.825147 10362 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:01:53.825515 10362 log.cc:1196] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 22:01:53.826078 10362 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: Deleting consensus metadata
I20260927 22:01:53.826427 10362 master_runner.cc:315] Copying system tablet from 127.3.159.62:33985
I20260927 22:01:53.829735 10362 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: tablet copy: Beginning tablet copy session from remote peer at address 127.3.159.62:33985
I20260927 22:01:53.853873 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.830918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46630)
0927 22:01:53.833917 (+ 2999us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:53.833946 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:53.834260 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:53.834908 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:53.834914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:53.835971 (+ 1057us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:53.836373 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.836389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.838354 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.838365 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.840047 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:53.840075 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.841581 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.841587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:53.841809 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.842815 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:53.842842 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:53.844036 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:53.849483 (+ 5447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.849490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.849495 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.850215 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.851105 (+ 890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:53.851114 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:53.851121 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:53.851367 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:53.852457 (+ 1090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:53.852464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:53.852703 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 22:01:53.852897 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 22:01:53.853279 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2804,"thread_start_us":2681,"threads_started":1}
I20260927 22:01:53.855213 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:53.831023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.3.159.62:33985)
0927 22:01:53.831273 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:53.831280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:53.834091 (+ 2811us) server_negotiation.cc:407] Connection header received
0927 22:01:53.834521 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:53.834531 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:53.834663 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:53.834848 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:53.836664 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.838025 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.840316 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:53.841087 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:53.841252 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:53.844407 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:53.844416 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:53.844422 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:53.844488 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:53.849236 (+ 4748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.850400 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.850407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.850411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.850839 (+ 428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:53.851547 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:53.851554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:53.851558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:53.852052 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:53.852218 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:53.854506 (+ 2288us) server_negotiation.cc:299] Negotiation successful
0927 22:01:53.854774 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:01:53.856756 9918 tablet_copy_service.cc:140] P 082117a1043840558566f1322649c58b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9ffa375e09fa444187a77ac58b5966ee ({username='oryx', principal='oryx/127.3.159.61@KRBTEST.COM'} at 127.0.0.1:46630)
I20260927 22:01:53.857079 9918 tablet_copy_service.cc:161] P 082117a1043840558566f1322649c58b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9ffa375e09fa444187a77ac58b5966ee at {username='oryx', principal='oryx/127.3.159.61@KRBTEST.COM'} at 127.0.0.1:46630: session id = 9ffa375e09fa444187a77ac58b5966ee-00000000000000000000000000000000
I20260927 22:01:53.861912 9918 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:01:53.865003 10362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:01:53.873030 10362 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: tablet copy: Starting download of 0 data blocks...
I20260927 22:01:53.873499 10362 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: tablet copy: Starting download of 1 WAL segments...
I20260927 22:01:53.877491 10362 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:01:53.883807 9918 tablet_copy_service.cc:342] P 082117a1043840558566f1322649c58b: Request end of tablet copy session 9ffa375e09fa444187a77ac58b5966ee-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.159.61@KRBTEST.COM'} at 127.0.0.1:46630
I20260927 22:01:53.884166 9918 tablet_copy_service.cc:433] P 082117a1043840558566f1322649c58b: ending tablet copy session 9ffa375e09fa444187a77ac58b5966ee-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9ffa375e09fa444187a77ac58b5966ee
I20260927 22:01:53.900034 10362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:53.903920 10472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:53.912652 10473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:53.913565 10362 server_base.cc:1034] running on GCE node
W20260927 22:01:53.914067 10475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:53.914762 10362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:53.915696 10362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:53.916982 10362 hybrid_clock.cc:648] HybridClock initialized: now 1790546513916892 us; error 74 us; skew 500 ppm
I20260927 22:01:53.917268 10362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:53 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546513, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:53.919852 10362 init.cc:377] Logged in from keytab as oryx/127.3.159.61@KRBTEST.COM (short username oryx)
I20260927 22:01:53.924260 10362 webserver.cc:460] Webserver started at http://127.3.159.61:43915/ using document root <none> and password file <none>
I20260927 22:01:53.924713 10362 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:53.924804 10362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:53.928561 10362 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 22:01:53.931054 10482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:53.931646 10362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 22:01:53.931830 10362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/wal
uuid: "9ffa375e09fa444187a77ac58b5966ee"
format_stamp: "Formatted at 2026-09-27 22:01:53 on dist-test-slave-sg4j"
server_key: "d8fb519fa818adafa732cda3bbce4c55"
server_key_iv: "8766fea4c1a55730f9d76c02c069fa4e"
server_key_version: "encryptionkey@0"
I20260927 22:01:53.932030 10362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:53.958779 10362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:53.964222 10362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:53.999630 10356 heartbeater.cc:503] Master 127.3.159.62:33985 was elected leader, sending a full tablet report...
I20260927 22:01:54.009784 10362 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.61:34347
I20260927 22:01:54.009840 10533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.61:34347 every 8 connection(s)
I20260927 22:01:54.010991 10362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
I20260927 22:01:54.016993 10534 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:01:54.021528 10534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: Bootstrap starting.
I20260927 22:01:54.045882 10534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:01:54.046626 10534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: Bootstrap complete.
I20260927 22:01:54.048055 10534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } }
I20260927 22:01:54.048398 10534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9ffa375e09fa444187a77ac58b5966ee, State: Initialized, Role: LEARNER
I20260927 22:01:54.048681 10534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } }
I20260927 22:01:54.050140 10534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:54.049882 10537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } } }
I20260927 22:01:54.050434 10537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: This master's current role is: LEARNER
I20260927 22:01:54.067492 10548 catalog_manager.cc:1259] Loaded cluster ID: 783c9c685ebe49c9a2ad6864a3e32582
I20260927 22:01:54.067634 10548 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: loading cluster ID for follower catalog manager: success
I20260927 22:01:54.071516 10548 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: acquiring CA information for follower catalog manager: success
I20260927 22:01:54.074457 10548 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 22:01:54.268513 10465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:54.268838 10465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:54.268997 10465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:54.276576 10465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:01:54.276706 10465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:54.276751 10465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:54.276847 10465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:01:54.276925 10465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:01:54.291266 10465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:33985,127.3.159.61:34347,127.3.159.60:34919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.60.keytab
--principal=oryx/127.3.159.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.60:34919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.159.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:54.293190 10465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:54.294929 10465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:54.310207 10556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:54.310325 10553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:54.310020 10554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:54.312983 10465 server_base.cc:1034] running on GCE node
I20260927 22:01:54.314173 10465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:54.316140 10465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:54.317435 10465 hybrid_clock.cc:648] HybridClock initialized: now 1790546514317321 us; error 52 us; skew 500 ppm
I20260927 22:01:54.317904 10465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:54 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546514, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:54.322279 10465 init.cc:377] Logged in from keytab as oryx/127.3.159.60@KRBTEST.COM (short username oryx)
I20260927 22:01:54.324467 10465 webserver.cc:460] Webserver started at http://127.3.159.60:43285/ using document root <none> and password file <none>
I20260927 22:01:54.325599 10465 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:54.325743 10465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:54.326133 10465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:01:54.329958 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data/instance:
uuid: "9653854c3fde4b5e9c6f0529cd7fa539"
format_stamp: "Formatted at 2026-09-27 22:01:54 on dist-test-slave-sg4j"
server_key: "a8c39fb2956d484be7f57e5f54c4e375"
server_key_iv: "f84d0cbf169bfc979fab258a33ba422c"
server_key_version: "encryptionkey@0"
I20260927 22:01:54.330837 10465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/wal/instance:
uuid: "9653854c3fde4b5e9c6f0529cd7fa539"
format_stamp: "Formatted at 2026-09-27 22:01:54 on dist-test-slave-sg4j"
server_key: "a8c39fb2956d484be7f57e5f54c4e375"
server_key_iv: "f84d0cbf169bfc979fab258a33ba422c"
server_key_version: "encryptionkey@0"
I20260927 22:01:54.338181 10465 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260927 22:01:54.342659 10563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:54.344650 10465 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:01:54.344864 10465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/wal
uuid: "9653854c3fde4b5e9c6f0529cd7fa539"
format_stamp: "Formatted at 2026-09-27 22:01:54 on dist-test-slave-sg4j"
server_key: "a8c39fb2956d484be7f57e5f54c4e375"
server_key_iv: "f84d0cbf169bfc979fab258a33ba422c"
server_key_version: "encryptionkey@0"
I20260927 22:01:54.345062 10465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:54.372010 10465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:54.378026 10465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:54.378337 10465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:54.407609 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.396409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:34347 (local address 127.0.0.1:32954)
0927 22:01:54.397924 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:54.397954 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:54.398225 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:54.398786 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:54.398792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:54.398819 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:54.399095 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.399114 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.401321 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.401329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.404855 (+ 3526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.404897 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.406529 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.406537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.406771 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.406908 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 22:01:54.407061 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 22:01:54.407235 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"thread_start_us":1149,"threads_started":1}
I20260927 22:01:54.407609 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.396876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.3.159.61:34347)
0927 22:01:54.397711 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:54.397718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:54.398125 (+ 407us) server_negotiation.cc:407] Connection header received
0927 22:01:54.398435 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:54.398451 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:54.398544 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:54.398689 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:54.399370 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.401070 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.405260 (+ 4190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.406258 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.406459 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.406754 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:54.407085 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 22:01:54.407232 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":373,"threads_started":1}
I20260927 22:01:54.408452 10465 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.60:34919
I20260927 22:01:54.408481 10617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.60:34919 every 8 connection(s)
I20260927 22:01:54.409135 10509 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 22:01:54.410043 10537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 082117a1043840558566f1322649c58b. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } } }
I20260927 22:01:54.410498 10537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: This master's current role is: LEARNER
I20260927 22:01:54.410837 10465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
I20260927 22:01:54.414109 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 10465
I20260927 22:01:54.416239 10618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:54.427676 10618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:01:54 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.159.60@KRBTEST.COM
Sep 27 22:01:54 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546514, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.60@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:01:54.445971 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.417354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.3.159.60:34919)
0927 22:01:54.418568 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:54.418587 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:54.418643 (+ 56us) server_negotiation.cc:407] Connection header received
0927 22:01:54.420648 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:54.420685 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:54.421581 (+ 896us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:54.422341 (+ 760us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:54.424101 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.426304 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.427895 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.429034 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.429265 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.435193 (+ 5928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:54.435204 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:54.435222 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:54.435280 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:54.439322 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.440311 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.440322 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.440332 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.440562 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.441015 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.441022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.441025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.442063 (+ 1038us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:54.442448 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:54.443930 (+ 1482us) server_negotiation.cc:299] Negotiation successful
0927 22:01:54.444404 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"spinlock_wait_cycles":37248,"thread_start_us":557,"threads_started":1}
I20260927 22:01:54.451277 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.431476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46636)
0927 22:01:54.432325 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:54.432438 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:54.432603 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:54.433439 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:54.433448 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:54.434397 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:54.434691 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.434700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.436212 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.436220 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.437998 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.438026 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.439031 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.439038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.439308 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.440138 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:54.440166 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:54.443680 (+ 3514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:54.447779 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.447799 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.447808 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.448452 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.448948 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.448953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.448957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.449136 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.449846 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:54.449983 (+ 137us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:54.450224 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 22:01:54.450434 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 22:01:54.450823 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":424,"threads_started":1}
I20260927 22:01:54.451320 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.431720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.3.159.62:33985)
0927 22:01:54.433056 (+ 1336us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:54.433060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:54.433076 (+ 16us) server_negotiation.cc:407] Connection header received
0927 22:01:54.433128 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:54.433132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:54.433195 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:54.433329 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:54.434881 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.436066 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.438250 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.438758 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.438943 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.443953 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:54.443959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:54.443964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:54.444012 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:54.447574 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.448614 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.448624 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.448628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.448823 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.449276 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.449281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.449285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.449668 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:54.449871 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:54.450547 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 22:01:54.450818 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1220,"thread_start_us":1088,"threads_started":1}
I20260927 22:01:54.453207 10618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:01:54 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546514, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.60@KRBTEST.COM for oryx/127.3.159.61@KRBTEST.COM
I20260927 22:01:54.467595 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.453946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:34347 (local address 127.0.0.1:32962)
0927 22:01:54.454139 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:54.454158 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:54.454258 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:54.454742 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:54.454747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:54.455143 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:54.455296 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.455305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.456392 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.456397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.457557 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.457577 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.458282 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.458288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.458481 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.459147 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:54.459165 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:54.461488 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:54.464823 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.464829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.464833 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.465267 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.465917 (+ 650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.465921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.465930 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.466078 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.466625 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:54.466629 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:54.466773 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 22:01:54.466915 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:01:54.467240 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 22:01:54.467604 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.454241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.3.159.61:34347)
0927 22:01:54.454468 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:54.454475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:54.454490 (+ 15us) server_negotiation.cc:407] Connection header received
0927 22:01:54.454539 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:54.454542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:54.454603 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:54.454720 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:54.455475 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.456242 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.457708 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.458116 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.458250 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.461708 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:54.461714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:54.461717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:54.461751 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:54.464624 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.465449 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.465455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.465458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.465740 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.466195 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.466201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.466204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.466491 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:54.466664 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:54.467006 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 22:01:54.467240 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 22:01:54.469022 10618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:01:54 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546514, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.60@KRBTEST.COM for oryx/127.3.159.60@KRBTEST.COM
I20260927 22:01:54.485301 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.470230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:34919 (local address 127.0.0.1:46036)
0927 22:01:54.470459 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:54.470487 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:54.470651 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:54.471224 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:54.471230 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:54.471688 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:54.471879 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.471888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.473053 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.473057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.473987 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.473999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.474776 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.474781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.474959 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.475686 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:54.475704 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:54.478598 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:54.482333 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.482339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.482342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.482875 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.483396 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.483400 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.483403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.483600 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.484265 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:54.484269 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:54.484455 (+ 186us) client_negotiation.cc:769] Sending connection context
0927 22:01:54.484616 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 22:01:54.484920 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 22:01:54.485708 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.470249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.3.159.60:34919)
0927 22:01:54.470468 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:54.470472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:54.470586 (+ 114us) server_negotiation.cc:407] Connection header received
0927 22:01:54.470891 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:54.470896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:54.470979 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:54.471140 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:54.472069 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.472883 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.474157 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.474599 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.474714 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.478841 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:54.478847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:54.478851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:54.478884 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:54.482072 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.483044 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.483049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.483052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.483241 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.483774 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.483779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.483781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.484126 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:54.484433 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:54.484752 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 22:01:54.485162 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":86}
I20260927 22:01:54.492256 10618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Bootstrap starting.
I20260927 22:01:54.496560 10618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:54.498072 10618 log.cc:824] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:54.503351 10618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: No bootstrap required, opened a new log
I20260927 22:01:54.512333 10618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } }
I20260927 22:01:54.512810 10618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:54.512910 10618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9653854c3fde4b5e9c6f0529cd7fa539, State: Initialized, Role: FOLLOWER
I20260927 22:01:54.513793 10618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } }
I20260927 22:01:54.515040 10625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } } }
I20260927 22:01:54.515676 10625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:01:54.516584 10618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 22:01:54.528807 10636 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:01:54.529040 10636 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:01:54.538390 10465 master_runner.cc:428] Detected that this master 9653854c3fde4b5e9c6f0529cd7fa539 is joining an existing cluster
I20260927 22:01:54.538570 10465 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.159.60:34919
I20260927 22:01:54.555248 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 9960
I20260927 22:01:54.566305 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.543746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.3.159.62:33985)
0927 22:01:54.544078 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:54.544087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:54.545498 (+ 1411us) server_negotiation.cc:407] Connection header received
0927 22:01:54.545858 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:54.545865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:54.545962 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:54.546049 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:54.548708 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.550090 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.552657 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.553424 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.553557 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.557322 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:54.557331 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:54.557337 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:54.557417 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:54.561515 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.562650 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.562658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.562662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.562865 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.563430 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.563435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.563439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.563869 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:54.564039 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:54.564731 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 22:01:54.565077 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 22:01:54.568148 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.543751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46642)
0927 22:01:54.545304 (+ 1553us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:54.545336 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:54.545652 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:54.546348 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:54.546356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:54.548023 (+ 1667us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:54.548462 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.548478 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.550425 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.550430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.552427 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.552446 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.553944 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.553951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.554143 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.555072 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:54.555100 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:54.556937 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:54.561838 (+ 4901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.561847 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.561856 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.562463 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.563028 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.563034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.563040 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.563274 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.564145 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:54.564150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:54.564404 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 22:01:54.567093 (+ 2689us) client_negotiation.cc:240] Negotiation successful
0927 22:01:54.567586 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1364,"thread_start_us":935,"threads_started":1}
I20260927 22:01:54.588534 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.544409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.3.159.60:34919)
0927 22:01:54.544719 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:54.544726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:54.568961 (+ 24235us) server_negotiation.cc:407] Connection header received
0927 22:01:54.569240 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:54.569246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:54.569323 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:54.569438 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:54.570726 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.571934 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.574013 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.574805 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.574924 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.578070 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:54.578079 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:54.578088 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:54.578123 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:54.584083 (+ 5960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.585563 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.585571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.585577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.585768 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.586218 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.586223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.586226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.586783 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:54.586940 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:54.587650 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 22:01:54.587978 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260927 22:01:54.590093 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.544617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:34919 (local address 127.0.0.1:46048)
0927 22:01:54.568808 (+ 24191us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:54.568837 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:54.569086 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:54.569666 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:54.569671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:54.570280 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:54.570537 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.570551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.572168 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.572174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.573823 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.573841 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.575268 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.575274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.575497 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.576357 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:54.576383 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:54.577836 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:54.584724 (+ 6888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.584731 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.584737 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.585412 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.585934 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.585939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.585942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.586097 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.587021 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:54.587026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:54.587229 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 22:01:54.589246 (+ 2017us) client_negotiation.cc:240] Negotiation successful
0927 22:01:54.589696 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24030,"threads_started":1}
Sep 27 22:01:54 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.159.1@KRBTEST.COM
Sep 27 22:01:54 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546510, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.62@KRBTEST.COM for oryx/127.3.159.60@KRBTEST.COM
I20260927 22:01:54.608737 10600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.545780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.3.159.61:34347)
0927 22:01:54.546015 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:54.546024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:54.591215 (+ 45191us) server_negotiation.cc:407] Connection header received
0927 22:01:54.599123 (+ 7908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:54.599131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:54.599414 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:54.599549 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:54.600492 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.601768 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.604655 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.605303 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.605460 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.606465 (+ 1005us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:54.607745 (+ 1280us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:54.607879 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:54.608286 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 22:01:54.608395 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:01:54.609771 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.546210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:34347 (local address 127.0.0.1:32968)
0927 22:01:54.591002 (+ 44792us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:54.591037 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:54.598744 (+ 7707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:54.599824 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:54.599829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:54.599846 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:01:54.600266 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.600275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.601996 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.602002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.604377 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.604401 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.605747 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.605755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.605998 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.606161 (+ 163us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:01:54.608074 (+ 1913us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:01:54.608100 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:01:54.609045 (+ 945us) client_negotiation.cc:240] Negotiation successful
0927 22:01:54.609224 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44409,"thread_start_us":35012,"threads_started":1}
I20260927 22:01:54.613511 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.590940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:34919 (local address 127.0.0.1:46060)
0927 22:01:54.591126 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:54.591149 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:54.591286 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:54.592074 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:54.592081 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:54.592684 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:54.592959 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.592974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.594868 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.594875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.596439 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.596461 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.597749 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.597757 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.597980 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.601725 (+ 3745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:54.601760 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:54.604624 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:54.610127 (+ 5503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.610135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.610139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.610752 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.611330 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.611336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.611340 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.611606 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.612403 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:54.612410 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:54.612616 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 22:01:54.612758 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:01:54.613100 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 22:01:54.613983 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.591357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46060 (local address 127.3.159.60:34919)
0927 22:01:54.591521 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:54.591527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:54.591544 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:01:54.591682 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:54.591688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:54.591826 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:54.593144 (+ 1318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:54.593417 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.594654 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.596712 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.597507 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.598451 (+ 944us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.604957 (+ 6506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:54.604964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:54.604969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:54.605014 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:54.609661 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.610951 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.610957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.610961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.611116 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.611774 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.611781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.611784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.612262 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:54.613258 (+ 996us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:54.613413 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 22:01:54.613679 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:01:54.615715 10645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.557825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34130 (local address 127.3.159.1:37633)
0927 22:01:54.572776 (+ 14951us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:54.572793 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:54.572852 (+ 59us) server_negotiation.cc:407] Connection header received
0927 22:01:54.572969 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:54.572984 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:54.573452 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:54.576607 (+ 3155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:54.576951 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.591905 (+ 14954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.595515 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.597338 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.598520 (+ 1182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.604477 (+ 5957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:54.604503 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:54.604527 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:54.604585 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:54.610105 (+ 5520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.611660 (+ 1555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.611673 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.611678 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.611854 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.612357 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.612364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.612371 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.612798 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:54.613119 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:54.614994 (+ 1875us) server_negotiation.cc:299] Negotiation successful
0927 22:01:54.615296 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14639,"thread_start_us":15842,"threads_started":1}
I20260927 22:01:54.616050 9898 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.159.60:34919: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539"
member_type: NON_VOTER
last_known_addr {
host: "127.3.159.60"
port: 34919
}
attrs {
promote: true
}
}
dest_uuid: "082117a1043840558566f1322649c58b"
cas_config_opid_index: 5
I20260927 22:01:54.617368 10083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:54.617491 9898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } }
I20260927 22:01:54.619911 10592 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9653854c3fde4b5e9c6f0529cd7fa539 (127.3.159.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
W20260927 22:01:54.631069 10569 tablet.cc:2369] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:54.632840 10592 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.159.60:34919
I20260927 22:01:54.633944 10465 master.cc:537] Master@127.3.159.60:34919 shutting down...
I20260927 22:01:54.641003 10509 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 LEARNER]: Refusing update from remote peer 082117a1043840558566f1322649c58b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 22:01:54.653923 10647 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 22:01:54.669020 10465 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:54.669715 10465 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:54.669986 10465 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: stopping tablet replica
I20260927 22:01:54.681506 10509 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9653854c3fde4b5e9c6f0529cd7fa539 (127.3.159.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
I20260927 22:01:54.691965 10648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
I20260927 22:01:54.692422 10648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: This master's current role is: LEADER
W20260927 22:01:54.693650 9885 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.159.60:34919, user_credentials={real_user=slave}} blocked reactor thread for 55880.8us
W20260927 22:01:54.694521 9885 consensus_peers.cc:487] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b -> Peer 9653854c3fde4b5e9c6f0529cd7fa539 (127.3.159.60:34919): Couldn't send request to peer 9653854c3fde4b5e9c6f0529cd7fa539. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:01:54.695237 10537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: NON_VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: true } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
I20260927 22:01:54.695664 10537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: This master's current role is: LEARNER
I20260927 22:01:54.697460 10592 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b: attempting to promote NON_VOTER 9ffa375e09fa444187a77ac58b5966ee to VOTER
I20260927 22:01:54.698621 10592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } }
I20260927 22:01:54.701932 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.701028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:34919 (local address 127.0.0.1:46072)
0927 22:01:54.701270 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:54.701457 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:34919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 22:01:54.702529 10509 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 LEARNER]: Refusing update from remote peer 082117a1043840558566f1322649c58b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 22:01:54.703363 10649 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 22:01:54.703994 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.703360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:34919 (local address 127.0.0.1:46076)
0927 22:01:54.703613 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:54.703724 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:34919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 22:01:54.704023 10465 master.cc:552] Master@127.3.159.60:34919 shutdown complete.
I20260927 22:01:54.704552 10465 master_runner.cc:305] Clearing existing system tablet
W20260927 22:01:54.704501 9885 consensus_peers.cc:487] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b -> Peer 9653854c3fde4b5e9c6f0529cd7fa539 (127.3.159.60:34919): Couldn't send request to peer 9653854c3fde4b5e9c6f0529cd7fa539. Status: Network error: Client connection negotiation failed: client connection to 127.3.159.60:34919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:01:54.705216 10465 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:54.705536 10465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:54.707135 10592 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9ffa375e09fa444187a77ac58b5966ee (127.3.159.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
I20260927 22:01:54.708221 10508 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9ffa375e09fa444187a77ac58b5966ee (127.3.159.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
I20260927 22:01:54.712352 10647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
I20260927 22:01:54.712600 10592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
I20260927 22:01:54.712910 10647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:54.712961 10592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:54.713141 10537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
I20260927 22:01:54.713536 10537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:01:54.722405 10465 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.003s sys 0.005s
I20260927 22:01:54.724865 10654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:54.725416 10465 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:54.725790 10465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 22:01:54.726094 10465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/wal
uuid: "9653854c3fde4b5e9c6f0529cd7fa539"
format_stamp: "Formatted at 2026-09-27 22:01:54 on dist-test-slave-sg4j"
server_key: "a8c39fb2956d484be7f57e5f54c4e375"
server_key_iv: "f84d0cbf169bfc979fab258a33ba422c"
server_key_version: "encryptionkey@0"
I20260927 22:01:54.731335 10465 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:54.740459 10465 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 22:01:54.740739 10465 log.cc:1196] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 22:01:54.741255 10465 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Deleting consensus metadata
I20260927 22:01:54.741596 10465 master_runner.cc:315] Copying system tablet from 127.3.159.62:33985
I20260927 22:01:54.748191 10465 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: tablet copy: Beginning tablet copy session from remote peer at address 127.3.159.62:33985
I20260927 22:01:54.762507 10660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.749147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46654)
0927 22:01:54.749783 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:54.749805 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:54.750001 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:54.750382 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:54.750386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:54.750946 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:54.751177 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.751189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.752227 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.752235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.753215 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:54.753226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.754022 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.754028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:54.754210 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.754958 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:54.754982 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:54.755885 (+ 903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:54.759283 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.759290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.759294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.759686 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.760078 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:54.760083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:54.760087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:54.760243 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:54.760822 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:54.760827 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:54.761498 (+ 671us) client_negotiation.cc:769] Sending connection context
0927 22:01:54.761652 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:01:54.762020 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":675,"threads_started":1}
I20260927 22:01:54.763090 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:54.749536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.3.159.62:33985)
0927 22:01:54.749787 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:54.749794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:54.750014 (+ 220us) server_negotiation.cc:407] Connection header received
0927 22:01:54.750136 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:54.750142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:54.750240 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:54.750360 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:54.751333 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.752080 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.753368 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:54.753848 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:54.753998 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:54.756121 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:54.756128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:54.756134 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:54.756174 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:54.759115 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.759821 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.759828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.759832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.759964 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:54.760348 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:54.760355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:54.760358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:54.760702 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:54.760866 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:54.761753 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 22:01:54.762329 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260927 22:01:54.763635 9918 tablet_copy_service.cc:140] P 082117a1043840558566f1322649c58b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9653854c3fde4b5e9c6f0529cd7fa539 ({username='oryx', principal='oryx/127.3.159.60@KRBTEST.COM'} at 127.0.0.1:46654)
I20260927 22:01:54.763808 9918 tablet_copy_service.cc:161] P 082117a1043840558566f1322649c58b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9653854c3fde4b5e9c6f0529cd7fa539 at {username='oryx', principal='oryx/127.3.159.60@KRBTEST.COM'} at 127.0.0.1:46654: session id = 9653854c3fde4b5e9c6f0529cd7fa539-00000000000000000000000000000000
I20260927 22:01:54.766213 9918 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 22:01:54.768548 10465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 22:01:54.774626 10465 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: tablet copy: Starting download of 0 data blocks...
I20260927 22:01:54.775460 10465 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: tablet copy: Starting download of 1 WAL segments...
I20260927 22:01:54.778043 10465 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 22:01:54.782015 9918 tablet_copy_service.cc:342] P 082117a1043840558566f1322649c58b: Request end of tablet copy session 9653854c3fde4b5e9c6f0529cd7fa539-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.159.60@KRBTEST.COM'} at 127.0.0.1:46654
I20260927 22:01:54.782186 9918 tablet_copy_service.cc:433] P 082117a1043840558566f1322649c58b: ending tablet copy session 9653854c3fde4b5e9c6f0529cd7fa539-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9653854c3fde4b5e9c6f0529cd7fa539
I20260927 22:01:54.794876 10465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:54.798050 10661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:54.798934 10662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:54.806773 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 9960
I20260927 22:01:54.829780 10465 server_base.cc:1034] running on GCE node
W20260927 22:01:54.831120 9882 connection.cc:535] server connection from 127.3.159.1:60303 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 22:01:54.831739 10664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:54.832384 10465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:54.833333 10465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:54.833292 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:37633
--local_ip_for_outbound_sockets=127.3.159.1
--tserver_master_addrs=127.3.159.62:33985
--webserver_port=32969
--webserver_interface=127.3.159.1
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab
--principal=oryx/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.159.62:33985,127.3.159.61:34347
--tserver_master_addrs=127.3.159.62:33985,127.3.159.61:34347,127.3.159.60:34919 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:54.834733 10465 hybrid_clock.cc:648] HybridClock initialized: now 1790546514834659 us; error 53 us; skew 500 ppm
I20260927 22:01:54.835141 10465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:54 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546514, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:54.837896 10465 init.cc:377] Logged in from keytab as oryx/127.3.159.60@KRBTEST.COM (short username oryx)
I20260927 22:01:54.841972 10465 webserver.cc:460] Webserver started at http://127.3.159.60:39873/ using document root <none> and password file <none>
I20260927 22:01:54.842423 10465 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:54.842538 10465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:54.845611 10465 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260927 22:01:54.860569 10672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:54.861105 10465 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.005s
I20260927 22:01:54.861259 10465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/wal
uuid: "9653854c3fde4b5e9c6f0529cd7fa539"
format_stamp: "Formatted at 2026-09-27 22:01:54 on dist-test-slave-sg4j"
server_key: "a8c39fb2956d484be7f57e5f54c4e375"
server_key_iv: "f84d0cbf169bfc979fab258a33ba422c"
server_key_version: "encryptionkey@0"
I20260927 22:01:54.861481 10465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:54.883311 10465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:54.887919 10465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:54.940656 10465 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.60:34919
I20260927 22:01:54.940659 10723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.60:34919 every 8 connection(s)
I20260927 22:01:54.942058 10465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
I20260927 22:01:54.948714 10724 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:01:54.953406 10724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Bootstrap starting.
I20260927 22:01:54.979415 10724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:01:54.980302 10724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Bootstrap complete.
I20260927 22:01:54.981789 10724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } }
I20260927 22:01:54.982180 10724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9653854c3fde4b5e9c6f0529cd7fa539, State: Initialized, Role: LEARNER
I20260927 22:01:54.982513 10724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } }
I20260927 22:01:54.984110 10724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:01:54.983865 10727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
I20260927 22:01:54.984486 10727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: This master's current role is: LEARNER
I20260927 22:01:54.995101 10738 catalog_manager.cc:1259] Loaded cluster ID: 783c9c685ebe49c9a2ad6864a3e32582
I20260927 22:01:54.995241 10738 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: loading cluster ID for follower catalog manager: success
I20260927 22:01:54.999356 10738 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: acquiring CA information for follower catalog manager: success
I20260927 22:01:55.002362 10738 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:01:55.223531 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:55.216376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46092 (local address 127.3.159.60:34919)
0927 22:01:55.217192 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:55.217198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:55.217216 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:01:55.217291 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:55.217296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:55.217444 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:55.217605 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:55.218120 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.219312 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.221057 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.222165 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.222346 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:55.222830 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:55.222896 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 22:01:55.223055 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":451,"threads_started":1}
I20260927 22:01:55.223519 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:55.216136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:34919 (local address 127.0.0.1:46092)
0927 22:01:55.216704 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:55.216740 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:55.216954 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:55.217624 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:55.217629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:55.217656 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:55.217945 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:55.217957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.219474 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.219482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:55.220876 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:55.220891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.222395 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.222402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:55.222630 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:55.222672 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:01:55.222839 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 22:01:55.223052 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":332,"threads_started":1}
I20260927 22:01:55.225163 10699 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 22:01:55.226227 10727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 082117a1043840558566f1322649c58b. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
I20260927 22:01:55.226683 10727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: This master's current role is: LEARNER
W20260927 22:01:55.296025 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:55.296355 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:55.296455 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:55.304385 10670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:55.304937 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:55.305104 10670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:01:55.316618 10670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab
--principal=oryx/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:37633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=32969
--tserver_master_addrs=127.3.159.62:33985,127.3.159.61:34347,127.3.159.60:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:55.318468 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:55.320125 10670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:55.340531 10748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:55.340560 10746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:55.340648 10745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:55.341084 10670 server_base.cc:1034] running on GCE node
I20260927 22:01:55.342347 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:55.343780 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:55.345062 10670 hybrid_clock.cc:648] HybridClock initialized: now 1790546515345005 us; error 43 us; skew 500 ppm
I20260927 22:01:55.345724 10670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:55 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546515, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:55.350126 10670 init.cc:377] Logged in from keytab as oryx/127.3.159.1@KRBTEST.COM (short username oryx)
I20260927 22:01:55.352358 10670 webserver.cc:460] Webserver started at http://127.3.159.1:32969/ using document root <none> and password file <none>
I20260927 22:01:55.353509 10670 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:55.353688 10670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:55.364464 10670 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.001s
I20260927 22:01:55.369585 10755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:55.371909 10670 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:01:55.372237 10670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "00d4d325d1014c8d9daeee0118430882"
format_stamp: "Formatted at 2026-09-27 22:01:51 on dist-test-slave-sg4j"
server_key: "dc00ff2a92bde921527f737d5aa908fb"
server_key_iv: "a295675d0bcf9319c09701cd8fb60964"
server_key_version: "encryptionkey@0"
I20260927 22:01:55.373752 10670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:55.400413 10670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:55.406859 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:55.407207 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:55.409094 10670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:55.411036 10670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:55.411144 10670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:55.411233 10670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:55.411294 10670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:55.459662 10670 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:37633
I20260927 22:01:55.459779 10867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:37633 every 8 connection(s)
I20260927 22:01:55.461606 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:01:55.465734 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 10670
I20260927 22:01:55.466060 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 10094
Sep 27 22:01:55 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546515, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.61@KRBTEST.COM
Sep 27 22:01:55 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546515, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.60@KRBTEST.COM
Sep 27 22:01:55 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546515, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
Sep 27 22:01:55 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.159.2@KRBTEST.COM
I20260927 22:01:55.507512 10740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:55.466769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:34425 (local address 127.3.159.60:34919)
0927 22:01:55.467017 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:55.467025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:55.483206 (+ 16181us) server_negotiation.cc:407] Connection header received
0927 22:01:55.483632 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:55.483639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:55.483737 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:55.483844 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:55.487576 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.488646 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.492065 (+ 3419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.492736 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.492832 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:55.498718 (+ 5886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:55.498726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:55.498731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:55.498774 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:55.502872 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:55.504645 (+ 1773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:55.504653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:55.504659 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:55.504870 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:55.505332 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:55.505337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:55.505340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:55.505745 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:55.505902 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:55.506695 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 22:01:55.506986 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 22:01:55.510504 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:55.466254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:34919 (local address 127.3.159.1:34425)
0927 22:01:55.483030 (+ 16776us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:55.483056 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:55.483460 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:55.484527 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:55.484540 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:55.485815 (+ 1275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:55.487377 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:55.487392 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.488942 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.488948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:55.491924 (+ 2976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:55.491935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.493138 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.493144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:55.493306 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:55.494100 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:55.494172 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:55.498459 (+ 4287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:55.503050 (+ 4591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:55.503054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:55.503069 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:55.504484 (+ 1415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:55.505021 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:55.505026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:55.505030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:55.505196 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:55.505978 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:55.505983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:55.506182 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 22:01:55.507844 (+ 1662us) client_negotiation.cc:240] Negotiation successful
0927 22:01:55.508216 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16554,"mutex_wait_us":798,"spinlock_wait_cycles":22400,"thread_start_us":21180,"threads_started":1}
I20260927 22:01:55.510541 10873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:55.466253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:34347 (local address 127.3.159.1:47061)
0927 22:01:55.477739 (+ 11486us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:55.477799 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:55.481645 (+ 3846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:55.485914 (+ 4269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:55.485920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:55.486380 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:55.486936 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:55.486974 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.489707 (+ 2733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.489713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:55.491642 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:55.491655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.492721 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.492728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:55.493049 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:55.494129 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:55.494180 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:55.497995 (+ 3815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:55.503017 (+ 5022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:55.503038 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:55.503070 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:55.503846 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:55.504322 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:55.504327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:55.504332 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:55.504711 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:55.505533 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:55.505543 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:55.506171 (+ 628us) client_negotiation.cc:769] Sending connection context
0927 22:01:55.506579 (+ 408us) client_negotiation.cc:240] Negotiation successful
0927 22:01:55.507318 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11071,"mutex_wait_us":25,"spinlock_wait_cycles":17152,"thread_start_us":10791,"threads_started":1}
I20260927 22:01:55.512671 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:55.478959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.1:49757)
0927 22:01:55.488684 (+ 9725us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:55.488706 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:55.488839 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:55.489685 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:55.489693 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:55.490761 (+ 1068us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:55.491212 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:55.491223 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.493702 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.493708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:55.495633 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:55.495658 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.496876 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.496883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:55.497125 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:55.498072 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:55.498096 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:55.503070 (+ 4974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:55.508888 (+ 5818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:55.508895 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:55.508899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:55.509556 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:55.510120 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:55.510127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:55.510132 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:55.510345 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:55.511201 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:55.511206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:55.511571 (+ 365us) client_negotiation.cc:769] Sending connection context
0927 22:01:55.511746 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 22:01:55.512142 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9585,"thread_start_us":3704,"threads_started":1}
I20260927 22:01:55.512826 10868 heartbeater.cc:347] Connected to a master server at 127.3.159.60:34919
I20260927 22:01:55.513026 10872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:55.465500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:47061 (local address 127.3.159.61:34347)
0927 22:01:55.485505 (+ 20005us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:55.485516 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:55.485545 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:01:55.485604 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:55.485609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:55.485695 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:55.486740 (+ 1045us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:55.487319 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.489537 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.491801 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.492457 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.496222 (+ 3765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:55.498364 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:55.498373 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:55.498379 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:55.498430 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:55.502739 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:55.503994 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:55.504000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:55.504004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:55.504171 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:55.504873 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:55.504879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:55.504882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:55.505348 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:55.510704 (+ 5356us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:55.510932 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 22:01:55.511271 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19788,"thread_start_us":19660,"threads_started":1}
I20260927 22:01:55.513265 10868 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:55.514281 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:55.478900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:49757 (local address 127.3.159.62:33985)
0927 22:01:55.488530 (+ 9630us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:55.488536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:55.488822 (+ 286us) server_negotiation.cc:407] Connection header received
0927 22:01:55.489050 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:55.489058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:55.489194 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:55.489482 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:55.491387 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.493505 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.495877 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.496628 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.496798 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:55.503477 (+ 6679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:55.503485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:55.503490 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:55.503540 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:55.508611 (+ 5071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:55.509757 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:55.509767 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:55.509772 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:55.509954 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:55.510523 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:55.510528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:55.510532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:55.510970 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:55.511118 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:55.513061 (+ 1943us) server_negotiation.cc:299] Negotiation successful
0927 22:01:55.513319 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9482,"thread_start_us":8248,"threads_started":1}
I20260927 22:01:55.514350 10870 heartbeater.cc:347] Connected to a master server at 127.3.159.61:34347
I20260927 22:01:55.514541 10868 heartbeater.cc:511] Master 127.3.159.60:34919 requested a full tablet report, sending...
I20260927 22:01:55.514541 10870 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:55.515269 10869 heartbeater.cc:347] Connected to a master server at 127.3.159.62:33985
I20260927 22:01:55.515308 10870 heartbeater.cc:511] Master 127.3.159.61:34347 requested a full tablet report, sending...
I20260927 22:01:55.515429 10869 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:55.516021 10869 heartbeater.cc:511] Master 127.3.159.62:33985 requested a full tablet report, sending...
I20260927 22:01:55.517779 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:55.467146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.3.159.2:33831)
0927 22:01:55.484189 (+ 17043us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:55.484201 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:55.484233 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:01:55.484302 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:55.484311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:55.484653 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:55.484827 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:55.488666 (+ 3839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.490713 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.494320 (+ 3607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:55.495446 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:55.495576 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:55.504270 (+ 8694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:55.504293 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:55.504315 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:55.504365 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:55.510526 (+ 6161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:55.513064 (+ 2538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:55.513076 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:55.513079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:55.513565 (+ 486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:55.515346 (+ 1781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:55.515351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:55.515354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:55.515865 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:55.516189 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:55.516816 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 22:01:55.517116 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16818,"thread_start_us":16660,"threads_started":1}
I20260927 22:01:55.517891 10499 ts_manager.cc:194] Registered new tserver with Master: 00d4d325d1014c8d9daeee0118430882 (127.3.159.1:37633)
I20260927 22:01:55.518308 10689 ts_manager.cc:194] Registered new tserver with Master: 00d4d325d1014c8d9daeee0118430882 (127.3.159.1:37633)
I20260927 22:01:55.519500 9898 ts_manager.cc:194] Re-registered known tserver with Master: 00d4d325d1014c8d9daeee0118430882 (127.3.159.1:37633)
I20260927 22:01:55.519501 10217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:55.520841 9898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:49757
I20260927 22:01:55.606837 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 10094
I20260927 22:01:55.626734 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:33831
--local_ip_for_outbound_sockets=127.3.159.2
--tserver_master_addrs=127.3.159.62:33985
--webserver_port=46185
--webserver_interface=127.3.159.2
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab
--principal=oryx/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.159.62:33985,127.3.159.61:34347
--tserver_master_addrs=127.3.159.62:33985,127.3.159.61:34347,127.3.159.60:34919 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:55.776417 10881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: NON_VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: true } } }
I20260927 22:01:55.776945 10881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:55.779701 10883 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b: attempting to promote NON_VOTER 9653854c3fde4b5e9c6f0529cd7fa539 to VOTER
I20260927 22:01:55.780619 10883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } }
I20260927 22:01:55.783857 10508 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 FOLLOWER]: Refusing update from remote peer 082117a1043840558566f1322649c58b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:01:55.784144 10699 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 1 LEARNER]: Refusing update from remote peer 082117a1043840558566f1322649c58b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 22:01:55.784739 10882 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:01:55.785462 10881 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 22:01:55.788941 10881 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9653854c3fde4b5e9c6f0529cd7fa539 (127.3.159.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:55.789724 10508 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9653854c3fde4b5e9c6f0529cd7fa539 (127.3.159.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:55.793723 10889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:55.794378 10889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:55.794749 10889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:55.795086 10889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:55.795408 10889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:55.795706 10889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:55.794902 10699 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9653854c3fde4b5e9c6f0529cd7fa539 (127.3.159.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:55.799497 10888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:55.800056 10888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:01:55.801899 10890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "082117a1043840558566f1322649c58b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:55.802219 10890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 22:01:56.070148 10880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:56.070482 10880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:56.070636 10880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:56.079382 10880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:56.079588 10880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:56.079794 10880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:01:56.095415 10880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab
--principal=oryx/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:33831
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=46185
--tserver_master_addrs=127.3.159.62:33985,127.3.159.61:34347,127.3.159.60:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:56.097286 10880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:56.099324 10880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:56.121639 10898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:56.121511 10896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:56.121443 10895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:56.121989 10880 server_base.cc:1034] running on GCE node
I20260927 22:01:56.123397 10880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:56.124627 10880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:56.128031 10880 hybrid_clock.cc:648] HybridClock initialized: now 1790546516127954 us; error 93 us; skew 500 ppm
I20260927 22:01:56.128563 10880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:56 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546516, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:56.134184 10880 init.cc:377] Logged in from keytab as oryx/127.3.159.2@KRBTEST.COM (short username oryx)
I20260927 22:01:56.136822 10880 webserver.cc:460] Webserver started at http://127.3.159.2:46185/ using document root <none> and password file <none>
I20260927 22:01:56.138166 10880 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:56.138402 10880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:56.147984 10880 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.001s
I20260927 22:01:56.152257 10905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:56.154932 10880 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 22:01:56.155165 10880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "11ff1411e4e14671a218145ee48c2962"
format_stamp: "Formatted at 2026-09-27 22:01:52 on dist-test-slave-sg4j"
server_key: "d38563af38b468b75845cf680e56f7cf"
server_key_iv: "0f3576356ae393b4ab2aa74f864bc664"
server_key_version: "encryptionkey@0"
I20260927 22:01:56.156364 10880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:56.193316 10880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:56.198624 10880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:56.198900 10880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:56.200073 10880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:56.202013 10880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:56.202121 10880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:56.202208 10880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:56.202268 10880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:56.243045 10880 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:33831
I20260927 22:01:56.243204 11017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:33831 every 8 connection(s)
I20260927 22:01:56.244697 10880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:01:56.251624 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 10880
I20260927 22:01:56.252039 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 10228
Sep 27 22:01:56 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.159.3@KRBTEST.COM
Sep 27 22:01:56 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546516, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.61@KRBTEST.COM
Sep 27 22:01:56 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546516, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.60@KRBTEST.COM
Sep 27 22:01:56 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546516, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:01:56.304370 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.254601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47976 (local address 127.3.159.3:34903)
0927 22:01:56.257421 (+ 2820us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:56.257439 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:56.270568 (+ 13129us) server_negotiation.cc:407] Connection header received
0927 22:01:56.271049 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:56.271061 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:56.271794 (+ 733us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:56.272085 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:56.275891 (+ 3806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.278209 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.281561 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.282601 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.282709 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:56.292839 (+ 10130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:56.292861 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:56.292885 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:56.292932 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:56.299184 (+ 6252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:56.300591 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:56.300605 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:56.300613 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:56.300815 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:56.301426 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:56.301435 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:56.301440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:56.301936 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:56.302283 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:56.303451 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0927 22:01:56.303846 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2438,"thread_start_us":2258,"threads_started":1}
I20260927 22:01:56.305765 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.249129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:49197 (local address 127.3.159.60:34919)
0927 22:01:56.281553 (+ 32424us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:56.281559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:56.281580 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:01:56.281641 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:56.281645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:56.281733 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:56.281840 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:56.284288 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.285671 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.288865 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.289732 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.289899 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:56.296707 (+ 6808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:56.296715 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:56.296721 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:56.296771 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:56.300978 (+ 4207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:56.302311 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:56.302317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:56.302320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:56.302481 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:56.303349 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:56.303355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:56.303358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:56.303770 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:56.303922 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:56.304999 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0927 22:01:56.305263 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32172,"thread_start_us":32002,"threads_started":1}
I20260927 22:01:56.307060 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.250275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:34919 (local address 127.3.159.2:49197)
0927 22:01:56.270380 (+ 20105us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:56.270457 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:56.274239 (+ 3782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:56.282088 (+ 7849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:56.282096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:56.282906 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:56.283430 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:56.283453 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.285941 (+ 2488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.285950 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:56.288550 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:56.288579 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.290557 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.290567 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:56.290836 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:56.292124 (+ 1288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:56.292156 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:56.296387 (+ 4231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:56.301181 (+ 4794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:56.301187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:56.301192 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:56.302139 (+ 947us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:56.302882 (+ 743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:56.302887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:56.302891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:56.303190 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:56.304040 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:56.304050 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:56.304580 (+ 530us) client_negotiation.cc:769] Sending connection context
0927 22:01:56.304898 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 22:01:56.305545 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19637,"mutex_wait_us":82,"thread_start_us":19235,"threads_started":1}
I20260927 22:01:56.307246 10351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:56.309152 11018 heartbeater.cc:347] Connected to a master server at 127.3.159.60:34919
I20260927 22:01:56.310431 11018 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:56.310902 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.249224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:49531 (local address 127.3.159.61:34347)
0927 22:01:56.263243 (+ 14019us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:56.263255 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:56.275968 (+ 12713us) server_negotiation.cc:407] Connection header received
0927 22:01:56.276062 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:56.276068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:56.276178 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:56.276341 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:56.280435 (+ 4094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.282099 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.287501 (+ 5402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.288564 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.288726 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:56.296426 (+ 7700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:56.296436 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:56.296440 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:56.296508 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:56.300839 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:56.302870 (+ 2031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:56.302880 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:56.302883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:56.303057 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:56.307980 (+ 4923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:56.307992 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:56.307999 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:56.308733 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:56.308972 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:56.309883 (+ 911us) server_negotiation.cc:299] Negotiation successful
0927 22:01:56.310266 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13757,"thread_start_us":14031,"threads_started":1}
I20260927 22:01:56.311709 11018 heartbeater.cc:511] Master 127.3.159.60:34919 requested a full tablet report, sending...
I20260927 22:01:56.312479 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.253572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:34347 (local address 127.3.159.2:49531)
0927 22:01:56.275641 (+ 22069us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:56.275669 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:56.275837 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:56.277139 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:56.277163 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:56.278675 (+ 1512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:56.279998 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:56.280072 (+ 74us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.282450 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.282460 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:56.287095 (+ 4635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:56.287137 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.289288 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.289299 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:56.289874 (+ 575us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:56.291018 (+ 1144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:56.291056 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:56.296037 (+ 4981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:56.301017 (+ 4980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:56.301029 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:56.301065 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:56.302737 (+ 1672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:56.307521 (+ 4784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:56.307527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:56.307531 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:56.307737 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:56.309074 (+ 1337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:56.309085 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:56.309341 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 22:01:56.311192 (+ 1851us) client_negotiation.cc:240] Negotiation successful
0927 22:01:56.311720 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21809,"mutex_wait_us":3,"thread_start_us":21545,"threads_started":1}
I20260927 22:01:56.314599 10689 ts_manager.cc:194] Registered new tserver with Master: 11ff1411e4e14671a218145ee48c2962 (127.3.159.2:33831)
I20260927 22:01:56.315217 11020 heartbeater.cc:347] Connected to a master server at 127.3.159.61:34347
I20260927 22:01:56.315450 11020 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:56.316325 11020 heartbeater.cc:511] Master 127.3.159.61:34347 requested a full tablet report, sending...
I20260927 22:01:56.316854 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.282127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.2:33099)
0927 22:01:56.288622 (+ 6495us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:56.288651 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:56.288829 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:56.291433 (+ 2604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:56.291438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:56.292076 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:56.292539 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:56.292554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.294522 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.294530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:56.296317 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:56.296330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.297679 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.297685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:56.297919 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:56.298549 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:56.298576 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:56.302370 (+ 3794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:56.306935 (+ 4565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:56.306940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:56.306944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:56.308687 (+ 1743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:56.314150 (+ 5463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:56.314158 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:56.314164 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:56.314409 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:56.315409 (+ 1000us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:56.315414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:56.315657 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 22:01:56.315827 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 22:01:56.316209 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6278,"thread_start_us":7984,"threads_started":1}
I20260927 22:01:56.316872 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.275990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:33099 (local address 127.3.159.62:33985)
0927 22:01:56.290820 (+ 14830us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:56.290830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:56.290859 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:01:56.290943 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:56.290949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:56.291066 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:56.291206 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:56.292937 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.294312 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.296489 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.297196 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.297348 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:56.302671 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:56.302681 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:56.302689 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:56.302792 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:56.306743 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:56.308849 (+ 2106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:56.308855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:56.308858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:56.313985 (+ 5127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:56.314562 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:56.314567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:56.314571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:56.315028 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:56.315193 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:56.316065 (+ 872us) server_negotiation.cc:299] Negotiation successful
0927 22:01:56.316334 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14604,"thread_start_us":16728,"threads_started":1}
I20260927 22:01:56.318401 10499 ts_manager.cc:194] Registered new tserver with Master: 11ff1411e4e14671a218145ee48c2962 (127.3.159.2:33831)
I20260927 22:01:56.319670 11019 heartbeater.cc:347] Connected to a master server at 127.3.159.62:33985
I20260927 22:01:56.319854 11019 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:56.320304 11019 heartbeater.cc:511] Master 127.3.159.62:33985 requested a full tablet report, sending...
I20260927 22:01:56.321588 9898 ts_manager.cc:194] Re-registered known tserver with Master: 11ff1411e4e14671a218145ee48c2962 (127.3.159.2:33831)
I20260927 22:01:56.322407 9898 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:33099
I20260927 22:01:56.406651 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 10228
I20260927 22:01:56.423838 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:34903
--local_ip_for_outbound_sockets=127.3.159.3
--tserver_master_addrs=127.3.159.62:33985
--webserver_port=44529
--webserver_interface=127.3.159.3
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab
--principal=oryx/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.159.62:33985,127.3.159.61:34347
--tserver_master_addrs=127.3.159.62:33985,127.3.159.61:34347,127.3.159.60:34919 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:01:56.525012 10869 heartbeater.cc:503] Master 127.3.159.62:33985 was elected leader, sending a full tablet report...
W20260927 22:01:56.816085 11031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:01:56.816402 11031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:01:56.816507 11031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:01:56.824167 11031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:01:56.824304 11031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:01:56.824422 11031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:01:56.835814 11031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab
--principal=oryx/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:34903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=44529
--tserver_master_addrs=127.3.159.62:33985,127.3.159.61:34347,127.3.159.60:34919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:01:56.837284 11031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:01:56.838841 11031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:01:56.856498 11037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:56.856630 11031 server_base.cc:1034] running on GCE node
W20260927 22:01:56.856653 11036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:01:56.856882 11039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:01:56.857940 11031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:01:56.859467 11031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:01:56.860821 11031 hybrid_clock.cc:648] HybridClock initialized: now 1790546516860697 us; error 109 us; skew 500 ppm
I20260927 22:01:56.861593 11031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:01:56 dist-test-slave-sg4j krb5kdc[9853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546516, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:01:56.865839 11031 init.cc:377] Logged in from keytab as oryx/127.3.159.3@KRBTEST.COM (short username oryx)
I20260927 22:01:56.867668 11031 webserver.cc:460] Webserver started at http://127.3.159.3:44529/ using document root <none> and password file <none>
I20260927 22:01:56.868424 11031 fs_manager.cc:359] Metadata directory not provided
I20260927 22:01:56.868549 11031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:01:56.875154 11031 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:01:56.879210 11046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:56.880928 11031 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260927 22:01:56.881096 11031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "60b8c7e9f16e49fb9a6538dec710976a"
format_stamp: "Formatted at 2026-09-27 22:01:52 on dist-test-slave-sg4j"
server_key: "42d7c33517d97e4bda4536be2f2525f7"
server_key_iv: "fdede79bc7ffe63c6a4c98b2101600d8"
server_key_version: "encryptionkey@0"
I20260927 22:01:56.882325 11031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:01:56.904284 11031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:01:56.911672 11031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:01:56.912022 11031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:01:56.913256 11031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:01:56.915179 11031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:01:56.915295 11031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:56.915469 11031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:01:56.915593 11031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:01:56.957806 11031 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:34903
I20260927 22:01:56.957924 11158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:34903 every 8 connection(s)
I20260927 22:01:56.959299 11031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:01:56.961297 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 11031
Sep 27 22:01:56 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546516, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.61@KRBTEST.COM
Sep 27 22:01:56 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546516, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.60@KRBTEST.COM
Sep 27 22:01:57 dist-test-slave-sg4j krb5kdc[9853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546516, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:01:57.024893 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.964326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:34347 (local address 127.3.159.3:56341)
0927 22:01:56.965860 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:56.965926 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:56.981601 (+ 15675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:56.983897 (+ 2296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:56.983905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:56.985886 (+ 1981us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:56.986370 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:56.986391 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.988895 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.988905 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:56.992480 (+ 3575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:56.992501 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.994180 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.994188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:56.994825 (+ 637us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:56.996466 (+ 1641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:56.996505 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:57.003236 (+ 6731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:57.019247 (+ 16011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:57.019264 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:57.019289 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:57.020073 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:57.020790 (+ 717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:57.020797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:57.020802 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:57.021092 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:57.021941 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:57.021952 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:57.022453 (+ 501us) client_negotiation.cc:769] Sending connection context
0927 22:01:57.022768 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 22:01:57.023429 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"mutex_wait_us":1142,"thread_start_us":777,"threads_started":1}
I20260927 22:01:57.026801 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.963871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:56341 (local address 127.3.159.61:34347)
0927 22:01:56.965012 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:56.965022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:56.966091 (+ 1069us) server_negotiation.cc:407] Connection header received
0927 22:01:56.983208 (+ 17117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:56.983221 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:56.983380 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:56.983505 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:56.986857 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.988448 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.992707 (+ 4259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.993484 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.993621 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.003597 (+ 9976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:57.003609 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:57.003618 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:57.003681 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:57.017876 (+ 14195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.020415 (+ 2539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.020425 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.020431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.020638 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.021329 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.021335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.021339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.021776 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:57.025930 (+ 4154us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.026083 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.026348 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":875,"threads_started":1}
I20260927 22:01:57.029119 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.965013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:42415 (local address 127.3.159.60:34919)
0927 22:01:56.979166 (+ 14153us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:56.979177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:56.982261 (+ 3084us) server_negotiation.cc:407] Connection header received
0927 22:01:56.982352 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:56.982361 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:56.982464 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:56.982590 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:56.986456 (+ 3866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.988161 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.992357 (+ 4196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.993576 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.993771 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.012158 (+ 18387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:57.012165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:57.012171 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:57.012222 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:57.018317 (+ 6095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.026225 (+ 7908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.026230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.026234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.026429 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.027162 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.027166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.027170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.027588 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:57.027730 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.028488 (+ 758us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.028746 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13958,"thread_start_us":13818,"threads_started":1}
I20260927 22:01:57.029235 11161 heartbeater.cc:347] Connected to a master server at 127.3.159.61:34347
I20260927 22:01:57.029870 11161 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:57.031111 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.964377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:34919 (local address 127.3.159.3:42415)
0927 22:01:56.981472 (+ 17095us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:56.981501 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:56.981747 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:56.983181 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:56.983202 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:56.984652 (+ 1450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:56.986090 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:56.986134 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.988449 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.988460 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:56.992007 (+ 3547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:56.992062 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.994555 (+ 2493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.994585 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:56.994935 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:56.996347 (+ 1412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:56.996406 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:57.011790 (+ 15384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:57.025440 (+ 13650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:57.025448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:57.025452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:57.026046 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:57.026591 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:57.026596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:57.026600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:57.026750 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:57.027818 (+ 1068us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:57.027822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:57.028153 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 22:01:57.029763 (+ 1610us) client_negotiation.cc:240] Negotiation successful
0927 22:01:57.030179 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16830,"mutex_wait_us":27,"thread_start_us":15910,"threads_started":1}
I20260927 22:01:57.032250 11161 heartbeater.cc:511] Master 127.3.159.61:34347 requested a full tablet report, sending...
I20260927 22:01:57.035627 11159 heartbeater.cc:347] Connected to a master server at 127.3.159.60:34919
I20260927 22:01:57.035787 11159 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:57.036303 11159 heartbeater.cc:511] Master 127.3.159.60:34919 requested a full tablet report, sending...
I20260927 22:01:57.038051 10689 ts_manager.cc:194] Registered new tserver with Master: 60b8c7e9f16e49fb9a6538dec710976a (127.3.159.3:34903)
I20260927 22:01:57.038585 10499 ts_manager.cc:194] Registered new tserver with Master: 60b8c7e9f16e49fb9a6538dec710976a (127.3.159.3:34903)
I20260927 22:01:57.044878 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.966920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.3:58213)
0927 22:01:56.992419 (+ 25499us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:56.992445 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:56.992724 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:56.993335 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:56.993341 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:56.994171 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:01:56.994798 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:56.994812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.996532 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:56.996540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.000842 (+ 4302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:57.000880 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.002967 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.002977 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.003347 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.004182 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:01:57.004217 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:01:57.021563 (+ 17346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:01:57.040432 (+ 18869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:57.040439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:57.040444 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:57.041009 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:57.042713 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:01:57.042719 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:01:57.042723 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:01:57.042924 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:01:57.043709 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:01:57.043717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:01:57.043915 (+ 198us) client_negotiation.cc:769] Sending connection context
0927 22:01:57.044080 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 22:01:57.044394 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25315,"mutex_wait_us":68,"thread_start_us":25099,"threads_started":1}
I20260927 22:01:57.044950 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.967251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:58213 (local address 127.3.159.62:33985)
0927 22:01:56.985395 (+ 18144us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:56.985405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:56.992581 (+ 7176us) server_negotiation.cc:407] Connection header received
0927 22:01:56.992876 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:56.992883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:56.992983 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:56.993076 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:56.995038 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:56.996295 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.001093 (+ 4798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.002145 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.002353 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.022055 (+ 19702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:57.022069 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:57.022079 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:57.022166 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:57.040207 (+ 18041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.042334 (+ 2127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.042340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.042345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.042570 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.043073 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.043078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.043081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.043571 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:57.043771 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.044257 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.044537 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17919,"thread_start_us":18147,"threads_started":1}
I20260927 22:01:57.047238 11160 heartbeater.cc:347] Connected to a master server at 127.3.159.62:33985
I20260927 22:01:57.047351 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.998807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.3.159.60:34919)
0927 22:01:56.999525 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:56.999549 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.001996 (+ 2447us) server_negotiation.cc:407] Connection header received
0927 22:01:57.005776 (+ 3780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.005785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.005898 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.006000 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:57.017751 (+ 11751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.019063 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.030495 (+ 11432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.031563 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.031668 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.036056 (+ 4388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:57.036063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:57.036067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:57.036113 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:57.039784 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.044559 (+ 4775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.044566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.044570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.044781 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.045324 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.045329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.045333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.045690 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:57.045922 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.046563 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.046825 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":347,"threads_started":1}
I20260927 22:01:57.047441 11160 heartbeater.cc:464] Registering TS with master...
I20260927 22:01:57.048007 11160 heartbeater.cc:511] Master 127.3.159.62:33985 requested a full tablet report, sending...
I20260927 22:01:57.049649 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:56.999005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.3.159.62:33985)
0927 22:01:57.005023 (+ 6018us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.005031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.005055 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:01:57.005129 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.005136 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.005240 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.009286 (+ 4046us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:57.009751 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.010966 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.016736 (+ 5770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.017529 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.030253 (+ 12724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.040090 (+ 9837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:57.040098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:57.040103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:57.040151 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:57.044134 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.045100 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.045106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.045109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.046171 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.046710 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.046714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.046718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.047107 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:57.047366 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.048922 (+ 1556us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.049183 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5811,"thread_start_us":5550,"threads_started":1}
I20260927 22:01:57.050977 9897 ts_manager.cc:194] Re-registered known tserver with Master: 60b8c7e9f16e49fb9a6538dec710976a (127.3.159.3:34903)
I20260927 22:01:57.052371 9897 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:58213
I20260927 22:01:57.065019 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.003361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43778 (local address 127.3.159.61:34347)
0927 22:01:57.027429 (+ 24068us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.027435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.043545 (+ 16110us) server_negotiation.cc:407] Connection header received
0927 22:01:57.045779 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.045786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.045893 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.045969 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:57.048127 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.049172 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.051183 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.051642 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.051720 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.056446 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:57.056455 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:57.056461 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:57.056515 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:57.061246 (+ 4731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.062550 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.062558 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.062563 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.062808 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.063344 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.063350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.063353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.063733 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:57.064058 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.064379 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.064645 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23945,"thread_start_us":38845,"threads_started":1}
I20260927 22:01:57.055917 9897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:01:57.067728 9897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:57.114619 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.096006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.3.159.2:33831)
0927 22:01:57.104550 (+ 8544us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.104561 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.104596 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:01:57.104728 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.104738 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.105083 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.105363 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:57.106224 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.107810 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.111745 (+ 3935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.113310 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.113542 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.113851 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.114051 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.114272 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8316,"thread_start_us":8171,"threads_started":1}
I20260927 22:01:57.114673 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.095779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:33831 (local address 127.0.0.1:42984)
0927 22:01:57.097541 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:57.097572 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:57.098218 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:57.105343 (+ 7125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:57.105351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:57.105412 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:57.105931 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:57.105944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.107960 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.107965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.111554 (+ 3589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:57.111566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.113618 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.113625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.113836 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.113881 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:01:57.113982 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:01:57.114161 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"mutex_wait_us":26,"thread_start_us":1256,"threads_started":1}
I20260927 22:01:57.115986 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.095928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:34903 (local address 127.0.0.1:47992)
0927 22:01:57.102116 (+ 6188us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:57.102146 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:57.102413 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:57.103395 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:57.103402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:57.103440 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:57.104830 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:57.104843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.107372 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.107379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.111137 (+ 3758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:57.111155 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.115029 (+ 3874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.115036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.115235 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.115271 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:01:57.115364 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 22:01:57.115521 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6013,"mutex_wait_us":22,"thread_start_us":5081,"threads_started":1}
I20260927 22:01:57.116767 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.098124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:37633 (local address 127.0.0.1:41768)
0927 22:01:57.104280 (+ 6156us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:57.104303 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:57.104534 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:57.105600 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:57.105604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:57.105638 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:57.106197 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:57.106207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.108384 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.108391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.111803 (+ 3412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:57.111826 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.115831 (+ 4005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.115838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.116087 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.116133 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:01:57.116229 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 22:01:57.116379 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5982,"mutex_wait_us":28,"thread_start_us":5700,"threads_started":1}
I20260927 22:01:57.117542 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.098355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.3.159.1:37633)
0927 22:01:57.098987 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.099000 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.104409 (+ 5409us) server_negotiation.cc:407] Connection header received
0927 22:01:57.104742 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.104750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.105312 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.105591 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:57.106394 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.108201 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.112054 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.115226 (+ 3172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.115397 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.115651 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.117037 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.117230 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":322,"threads_started":1}
I20260927 22:01:57.119191 10953 tablet_service.cc:1467] Processing CreateTablet for tablet fc958e7aef994a9297f527dda74c5bda (DEFAULT_TABLE table=test-table [id=5e51dedf3794446485b1c7c118f0cf01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:57.120752 10953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc958e7aef994a9297f527dda74c5bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:57.122539 10803 tablet_service.cc:1467] Processing CreateTablet for tablet fc958e7aef994a9297f527dda74c5bda (DEFAULT_TABLE table=test-table [id=5e51dedf3794446485b1c7c118f0cf01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:57.123728 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.097478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47992 (local address 127.3.159.3:34903)
0927 22:01:57.102400 (+ 4922us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.102411 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.102465 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:01:57.102626 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.102640 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.103049 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.103382 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:57.105084 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.107172 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.111405 (+ 4233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.113226 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.114377 (+ 1151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.114766 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.115754 (+ 988us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.115971 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4636,"thread_start_us":4533,"threads_started":1}
I20260927 22:01:57.124423 10803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc958e7aef994a9297f527dda74c5bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:57.129120 11094 tablet_service.cc:1467] Processing CreateTablet for tablet fc958e7aef994a9297f527dda74c5bda (DEFAULT_TABLE table=test-table [id=5e51dedf3794446485b1c7c118f0cf01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:57.131325 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc958e7aef994a9297f527dda74c5bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:57.136950 11196 tablet_bootstrap.cc:492] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: Bootstrap starting.
I20260927 22:01:57.141243 11196 tablet_bootstrap.cc:654] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:57.143205 11196 log.cc:824] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:57.146795 11196 tablet_bootstrap.cc:492] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: No bootstrap required, opened a new log
I20260927 22:01:57.147142 11196 ts_tablet_manager.cc:1397] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 22:01:57.147666 11198 tablet_bootstrap.cc:492] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: Bootstrap starting.
I20260927 22:01:57.152726 11198 tablet_bootstrap.cc:654] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:57.154302 11198 log.cc:824] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:57.158553 11199 tablet_bootstrap.cc:492] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: Bootstrap starting.
I20260927 22:01:57.162534 11196 raft_consensus.cc:348] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } }
I20260927 22:01:57.163165 11196 raft_consensus.cc:374] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:57.163288 11196 raft_consensus.cc:729] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ff1411e4e14671a218145ee48c2962, State: Initialized, Role: FOLLOWER
I20260927 22:01:57.164084 11196 consensus_queue.cc:260] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } }
I20260927 22:01:57.164294 11199 tablet_bootstrap.cc:654] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:57.165772 11198 tablet_bootstrap.cc:492] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: No bootstrap required, opened a new log
I20260927 22:01:57.166143 11198 ts_tablet_manager.cc:1397] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20260927 22:01:57.166637 11019 heartbeater.cc:503] Master 127.3.159.62:33985 was elected leader, sending a full tablet report...
I20260927 22:01:57.167083 11199 log.cc:824] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:01:57.166290 11196 ts_tablet_manager.cc:1428] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: Time spent starting tablet: real 0.019s user 0.016s sys 0.004s
I20260927 22:01:57.173302 11199 tablet_bootstrap.cc:492] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: No bootstrap required, opened a new log
I20260927 22:01:57.173780 11199 ts_tablet_manager.cc:1397] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20260927 22:01:57.180436 11198 raft_consensus.cc:348] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } }
I20260927 22:01:57.180833 11198 raft_consensus.cc:374] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:57.180963 11198 raft_consensus.cc:729] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d4d325d1014c8d9daeee0118430882, State: Initialized, Role: FOLLOWER
I20260927 22:01:57.181697 11198 consensus_queue.cc:260] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } }
I20260927 22:01:57.183590 11198 ts_tablet_manager.cc:1428] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: Time spent starting tablet: real 0.017s user 0.013s sys 0.001s
I20260927 22:01:57.184983 11199 raft_consensus.cc:348] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } }
I20260927 22:01:57.185640 11199 raft_consensus.cc:374] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:57.185753 11199 raft_consensus.cc:729] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b8c7e9f16e49fb9a6538dec710976a, State: Initialized, Role: FOLLOWER
I20260927 22:01:57.186622 11199 consensus_queue.cc:260] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } }
I20260927 22:01:57.190892 11199 ts_tablet_manager.cc:1428] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260927 22:01:57.191314 11160 heartbeater.cc:503] Master 127.3.159.62:33985 was elected leader, sending a full tablet report...
W20260927 22:01:57.213433 11162 tablet.cc:2369] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:57.213310 11201 raft_consensus.cc:482] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:57.213776 11201 raft_consensus.cc:504] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } }
I20260927 22:01:57.215969 11201 leader_election.cc:288] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b8c7e9f16e49fb9a6538dec710976a (127.3.159.3:34903), 00d4d325d1014c8d9daeee0118430882 (127.3.159.1:37633)
W20260927 22:01:57.216904 10871 tablet.cc:2369] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:57.226648 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.217584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:50827 (local address 127.3.159.3:34903)
0927 22:01:57.217879 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.217888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.220154 (+ 2266us) server_negotiation.cc:407] Connection header received
0927 22:01:57.220386 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.220392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.220510 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.220638 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:57.221588 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.222366 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.224571 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.225448 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.225617 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.225730 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.226192 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.226342 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 22:01:57.226922 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.216458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:37633 (local address 127.3.159.2:33645)
0927 22:01:57.219826 (+ 3368us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:57.219853 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:57.220059 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:57.220580 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:57.220588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:57.220637 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:57.221096 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:57.221109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.222400 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.222408 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.224208 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:57.224224 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.225669 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.225676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.225951 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.226022 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 22:01:57.226157 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:01:57.226424 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3120,"mutex_wait_us":94,"thread_start_us":3033,"threads_started":1}
I20260927 22:01:57.227195 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.217115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:33645 (local address 127.3.159.1:37633)
0927 22:01:57.217446 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.217455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.220020 (+ 2565us) server_negotiation.cc:407] Connection header received
0927 22:01:57.220309 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.220317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.220414 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.220490 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:57.221282 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.222216 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.224421 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.225314 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.225558 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.225700 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.226210 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.226379 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 22:01:57.227386 11205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.216458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:34903 (local address 127.3.159.2:50827)
0927 22:01:57.219928 (+ 3470us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:57.219975 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:57.220228 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:57.220791 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:57.220796 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:57.220830 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:57.221307 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:57.221318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.222540 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.222544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.224278 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:57.224294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.225735 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.225742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:57.225986 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.226029 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:01:57.226160 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 22:01:57.226441 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3333,"mutex_wait_us":251,"thread_start_us":1539,"threads_started":1}
I20260927 22:01:57.228502 10823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc958e7aef994a9297f527dda74c5bda" candidate_uuid: "11ff1411e4e14671a218145ee48c2962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00d4d325d1014c8d9daeee0118430882" is_pre_election: true
I20260927 22:01:57.228825 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc958e7aef994a9297f527dda74c5bda" candidate_uuid: "11ff1411e4e14671a218145ee48c2962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b8c7e9f16e49fb9a6538dec710976a" is_pre_election: true
I20260927 22:01:57.229131 10823 raft_consensus.cc:2454] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11ff1411e4e14671a218145ee48c2962 in term 0.
I20260927 22:01:57.229590 11114 raft_consensus.cc:2454] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11ff1411e4e14671a218145ee48c2962 in term 0.
I20260927 22:01:57.230274 10906 leader_election.cc:302] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d4d325d1014c8d9daeee0118430882, 11ff1411e4e14671a218145ee48c2962; no voters:
I20260927 22:01:57.231770 11201 raft_consensus.cc:2781] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:57.232015 11201 raft_consensus.cc:482] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:57.232183 11201 raft_consensus.cc:3037] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:57.234953 11201 raft_consensus.cc:504] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } }
I20260927 22:01:57.235668 11201 leader_election.cc:288] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [CANDIDATE]: Term 1 election: Requested vote from peers 60b8c7e9f16e49fb9a6538dec710976a (127.3.159.3:34903), 00d4d325d1014c8d9daeee0118430882 (127.3.159.1:37633)
I20260927 22:01:57.236322 10823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc958e7aef994a9297f527dda74c5bda" candidate_uuid: "11ff1411e4e14671a218145ee48c2962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00d4d325d1014c8d9daeee0118430882"
I20260927 22:01:57.236549 10823 raft_consensus.cc:3037] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:57.236541 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc958e7aef994a9297f527dda74c5bda" candidate_uuid: "11ff1411e4e14671a218145ee48c2962" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b8c7e9f16e49fb9a6538dec710976a"
I20260927 22:01:57.236897 11114 raft_consensus.cc:3037] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:57.239018 10823 raft_consensus.cc:2454] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11ff1411e4e14671a218145ee48c2962 in term 1.
I20260927 22:01:57.239626 10906 leader_election.cc:302] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d4d325d1014c8d9daeee0118430882, 11ff1411e4e14671a218145ee48c2962; no voters:
I20260927 22:01:57.239840 11114 raft_consensus.cc:2454] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11ff1411e4e14671a218145ee48c2962 in term 1.
I20260927 22:01:57.240106 11201 raft_consensus.cc:2781] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:57.240686 11201 raft_consensus.cc:686] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 1 LEADER]: Becoming Leader. State: Replica: 11ff1411e4e14671a218145ee48c2962, State: Running, Role: LEADER
I20260927 22:01:57.241314 11201 consensus_queue.cc:237] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } }
I20260927 22:01:57.247834 9897 catalog_manager.cc:5495] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11ff1411e4e14671a218145ee48c2962 (127.3.159.2). New cstate: current_term: 1 leader_uuid: "11ff1411e4e14671a218145ee48c2962" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } health_report { overall_health: UNKNOWN } } }
W20260927 22:01:57.255317 11021 tablet.cc:2369] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:01:57.283999 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.277233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43000 (local address 127.3.159.2:33831)
0927 22:01:57.277581 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.277588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.277614 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:01:57.277843 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.277847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.277989 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.278298 (+ 309us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:57.278822 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.279718 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.281209 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.281719 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.281883 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.282517 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:57.283270 (+ 753us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:57.283490 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.283548 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.283653 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 22:01:57.295514 11114 raft_consensus.cc:1261] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a [term 1 FOLLOWER]: Refusing update from remote peer 11ff1411e4e14671a218145ee48c2962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:57.295563 10823 raft_consensus.cc:1261] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882 [term 1 FOLLOWER]: Refusing update from remote peer 11ff1411e4e14671a218145ee48c2962: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:57.296904 11207 consensus_queue.cc:1035] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:57.297495 11201 consensus_queue.cc:1035] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:57.344533 9897 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:57.344853 9897 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:57.352779 9897 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 082117a1043840558566f1322649c58b: Sending DeleteTablet for 3 replicas of tablet fc958e7aef994a9297f527dda74c5bda
I20260927 22:01:57.354139 11094 tablet_service.cc:1514] Processing DeleteTablet for tablet fc958e7aef994a9297f527dda74c5bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:57 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:47992
I20260927 22:01:57.354226 10953 tablet_service.cc:1514] Processing DeleteTablet for tablet fc958e7aef994a9297f527dda74c5bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:57 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:42984
I20260927 22:01:57.356083 10803 tablet_service.cc:1514] Processing DeleteTablet for tablet fc958e7aef994a9297f527dda74c5bda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:57 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:41768
I20260927 22:01:57.366168 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 9866
I20260927 22:01:57.370057 9928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:57.370236 11220 tablet_replica.cc:330] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: stopping tablet replica
I20260927 22:01:57.371052 11220 raft_consensus.cc:2229] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:57.371932 11220 raft_consensus.cc:2258] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:57.375030 11222 tablet_replica.cc:330] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: stopping tablet replica
I20260927 22:01:57.376086 11222 raft_consensus.cc:2229] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:57.377085 11222 raft_consensus.cc:2258] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:57.378549 11221 tablet_replica.cc:330] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: stopping tablet replica
I20260927 22:01:57.379323 11221 raft_consensus.cc:2229] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:01:57.381191 11221 raft_consensus.cc:2258] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:57.381578 11222 ts_tablet_manager.cc:1905] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:57.382786 11220 ts_tablet_manager.cc:1905] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:57.384117 11221 ts_tablet_manager.cc:1905] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:57.388000 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.360989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.3.159.61:34347)
0927 22:01:57.361305 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.361315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.364019 (+ 2704us) server_negotiation.cc:407] Connection header received
0927 22:01:57.364254 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.364263 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.364400 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.364526 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:57.367649 (+ 3123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.369234 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.372121 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.372686 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.372832 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.378797 (+ 5965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:57.378804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:57.378808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:57.378862 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:57.383002 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.384181 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.384187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.384192 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.384361 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.384954 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.384962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.384967 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.385430 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:57.385720 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.386981 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.387349 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 22:01:57.390918 11222 ts_tablet_manager.cc:1918] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:57.391188 11222 log.cc:1196] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/fc958e7aef994a9297f527dda74c5bda
I20260927 22:01:57.392100 11222 ts_tablet_manager.cc:1939] T fc958e7aef994a9297f527dda74c5bda P 00d4d325d1014c8d9daeee0118430882: Deleting consensus metadata
I20260927 22:01:57.392066 11220 ts_tablet_manager.cc:1918] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:57.392403 11220 log.cc:1196] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/fc958e7aef994a9297f527dda74c5bda
I20260927 22:01:57.393078 11220 ts_tablet_manager.cc:1939] T fc958e7aef994a9297f527dda74c5bda P 60b8c7e9f16e49fb9a6538dec710976a: Deleting consensus metadata
I20260927 22:01:57.395063 9882 catalog_manager.cc:4841] TS 00d4d325d1014c8d9daeee0118430882 (127.3.159.1:37633): tablet fc958e7aef994a9297f527dda74c5bda (table test-table [id=5e51dedf3794446485b1c7c118f0cf01]) successfully deleted
I20260927 22:01:57.405530 11221 ts_tablet_manager.cc:1918] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:57.405768 11221 log.cc:1196] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/fc958e7aef994a9297f527dda74c5bda
I20260927 22:01:57.406625 11221 ts_tablet_manager.cc:1939] T fc958e7aef994a9297f527dda74c5bda P 11ff1411e4e14671a218145ee48c2962: Deleting consensus metadata
I20260927 22:01:57.409955 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.365125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.3.159.60:34919)
0927 22:01:57.365435 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.365443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.390127 (+ 24684us) server_negotiation.cc:407] Connection header received
0927 22:01:57.390385 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.390391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.390477 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.390563 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:57.392784 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.394023 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.396387 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.397056 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.397172 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.401722 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:57.401730 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:57.401734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:57.401795 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:57.405512 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.406744 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.406753 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.406757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.406946 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.407466 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.407471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.407475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.407914 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:57.408164 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.409197 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.409567 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:01:57.523775 9885 catalog_manager.cc:4841] TS 11ff1411e4e14671a218145ee48c2962 (127.3.159.2:33831): tablet fc958e7aef994a9297f527dda74c5bda (table test-table [id=5e51dedf3794446485b1c7c118f0cf01]) successfully deleted
I20260927 22:01:57.524827 9884 catalog_manager.cc:4841] TS 60b8c7e9f16e49fb9a6538dec710976a (127.3.159.3:34903): tablet fc958e7aef994a9297f527dda74c5bda (table test-table [id=5e51dedf3794446485b1c7c118f0cf01]) successfully deleted
W20260927 22:01:57.525017 9884 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.3.159.3:34903, user_credentials={real_user=slave}} blocked reactor thread for 128351us
I20260927 22:01:57.525867 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 9866
I20260927 22:01:57.578675 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.556757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.3.159.60:34919)
0927 22:01:57.557010 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.557018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.558230 (+ 1212us) server_negotiation.cc:407] Connection header received
0927 22:01:57.558600 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.558608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.558713 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.558799 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:57.561528 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.562797 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.565104 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.565866 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.565982 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.570410 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:57.570418 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:57.570423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:57.570479 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:57.574816 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.575714 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.575720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.575725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.575957 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.576579 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.576584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.576588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.577029 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:57.577323 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.577829 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.578257 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 22:01:57.581625 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:57.558041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.3.159.61:34347)
0927 22:01:57.558371 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:57.558384 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:57.565110 (+ 6726us) server_negotiation.cc:407] Connection header received
0927 22:01:57.565999 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:57.566008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:57.566129 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:57.566228 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:57.568758 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.569663 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.571475 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:57.571920 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:57.572074 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:57.574798 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:57.574804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:57.574807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:57.574845 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:57.578275 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.579206 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.579215 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.579219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.579442 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:57.579937 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:57.579943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:57.579947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:57.580292 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:57.580578 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:57.581002 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 22:01:57.581242 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 22:01:58.327394 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.326424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.1:49981)
0927 22:01:58.327067 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.327212 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":349,"threads_started":1}
I20260927 22:01:58.327579 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.326289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.3:35967)
0927 22:01:58.326967 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.327170 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":323,"threads_started":1}
W20260927 22:01:58.327931 10869 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:33985 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
W20260927 22:01:58.328172 11160 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:33985 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
I20260927 22:01:58.334021 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.332945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.2:43449)
0927 22:01:58.333615 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.333786 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":322,"threads_started":1}
W20260927 22:01:58.334554 11019 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:33985 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
I20260927 22:01:58.853962 11235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 082117a1043840558566f1322649c58b)
I20260927 22:01:58.854348 11236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 082117a1043840558566f1322649c58b)
I20260927 22:01:58.854393 11235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } }
I20260927 22:01:58.854729 11236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } }
I20260927 22:01:58.856292 11235 leader_election.cc:288] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 082117a1043840558566f1322649c58b (127.3.159.62:33985), 9653854c3fde4b5e9c6f0529cd7fa539 (127.3.159.60:34919)
I20260927 22:01:58.856478 11236 leader_election.cc:288] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 082117a1043840558566f1322649c58b (127.3.159.62:33985), 9ffa375e09fa444187a77ac58b5966ee (127.3.159.61:34347)
I20260927 22:01:58.858297 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.856378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:45950)
0927 22:01:58.857554 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.857882 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":701,"threads_started":1}
W20260927 22:01:58.858728 10483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
I20260927 22:01:58.860283 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.856845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:45964)
0927 22:01:58.859680 (+ 2835us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.859934 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2643,"thread_start_us":2434,"threads_started":1}
W20260927 22:01:58.871009 10673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
I20260927 22:01:58.872918 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.859870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:34347 (local address 127.0.0.1:43804)
0927 22:01:58.860612 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.860637 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:58.860824 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:58.862138 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:58.862147 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:58.862202 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:58.862700 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:58.862720 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.865337 (+ 2617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.865346 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:58.868676 (+ 3330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:58.868711 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.871495 (+ 2784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.871506 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:58.871854 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:58.871926 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 22:01:58.872031 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 22:01:58.872347 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":10718,"threads_started":1}
I20260927 22:01:58.875347 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.859941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.3.159.61:34347)
0927 22:01:58.861635 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:58.861644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:58.861678 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:01:58.861778 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:58.861788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:58.861918 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:58.862892 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:58.863216 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.865072 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.869049 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.871099 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.873354 (+ 2255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:58.873585 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:58.873667 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 22:01:58.873908 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1407,"thread_start_us":1155,"threads_started":1}
I20260927 22:01:58.876271 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.875748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:45994)
0927 22:01:58.875926 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.876048 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 22:01:58.876240 10508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9ffa375e09fa444187a77ac58b5966ee" is_pre_election: true
W20260927 22:01:58.876628 10673 leader_election.cc:334] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 082117a1043840558566f1322649c58b (127.3.159.62:33985): Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
I20260927 22:01:58.876809 10508 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9653854c3fde4b5e9c6f0529cd7fa539 in term 1.
I20260927 22:01:58.877166 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.868467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:45980)
0927 22:01:58.876248 (+ 7781us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.876635 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7580}
W20260927 22:01:58.877416 10483 leader_election.cc:334] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 082117a1043840558566f1322649c58b (127.3.159.62:33985): Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
I20260927 22:01:58.877672 10673 leader_election.cc:302] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9653854c3fde4b5e9c6f0529cd7fa539, 9ffa375e09fa444187a77ac58b5966ee; no voters: 082117a1043840558566f1322649c58b
I20260927 22:01:58.878120 11236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 22:01:58.878315 11236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 082117a1043840558566f1322649c58b)
I20260927 22:01:58.878456 11236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:01:58.879549 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.856720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:34919 (local address 127.0.0.1:55736)
0927 22:01:58.858646 (+ 1926us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.858680 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:58.858923 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:58.870920 (+ 11997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:58.870926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:58.870979 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:58.871448 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:58.871463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.874414 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.874427 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:58.876864 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:58.876883 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.878763 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.878770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:58.878964 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:58.879011 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 22:01:58.879137 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 22:01:58.879317 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1771,"thread_start_us":20184,"threads_started":1}
I20260927 22:01:58.879920 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.857517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55736 (local address 127.3.159.60:34919)
0927 22:01:58.869988 (+ 12471us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:58.869995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:58.870024 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:01:58.870140 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:58.870150 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:58.870274 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:58.871642 (+ 1368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:58.872104 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.873793 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.877105 (+ 3312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.878467 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.878721 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:58.878893 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:58.879204 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 22:01:58.879398 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12237,"thread_start_us":13214,"threads_started":1}
I20260927 22:01:58.880636 10699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ffa375e09fa444187a77ac58b5966ee" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" is_pre_election: true
I20260927 22:01:58.882120 11236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } }
I20260927 22:01:58.882804 10699 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ffa375e09fa444187a77ac58b5966ee in current term 2: Already voted for candidate 9653854c3fde4b5e9c6f0529cd7fa539 in this term.
I20260927 22:01:58.883059 11236 leader_election.cc:288] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [CANDIDATE]: Term 2 election: Requested vote from peers 082117a1043840558566f1322649c58b (127.3.159.62:33985), 9ffa375e09fa444187a77ac58b5966ee (127.3.159.61:34347)
I20260927 22:01:58.883713 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.883232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46004)
0927 22:01:58.883384 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.883528 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 22:01:58.884019 10508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9ffa375e09fa444187a77ac58b5966ee"
I20260927 22:01:58.884132 10486 leader_election.cc:302] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ffa375e09fa444187a77ac58b5966ee; no voters: 082117a1043840558566f1322649c58b, 9653854c3fde4b5e9c6f0529cd7fa539
I20260927 22:01:58.884337 10508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:01:58.885366 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.884969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46012)
0927 22:01:58.885165 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.885239 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 22:01:58.885571 10673 leader_election.cc:334] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 082117a1043840558566f1322649c58b (127.3.159.62:33985): Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
I20260927 22:01:58.887379 10508 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9653854c3fde4b5e9c6f0529cd7fa539 in term 2.
I20260927 22:01:58.887754 11235 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 22:01:58.888095 10673 leader_election.cc:302] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9653854c3fde4b5e9c6f0529cd7fa539, 9ffa375e09fa444187a77ac58b5966ee; no voters: 082117a1043840558566f1322649c58b
I20260927 22:01:58.888533 11236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:01:58.889154 11236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [term 2 LEADER]: Becoming Leader. State: Replica: 9653854c3fde4b5e9c6f0529cd7fa539, State: Running, Role: LEADER
I20260927 22:01:58.889735 11236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } }
I20260927 22:01:58.893100 11245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9653854c3fde4b5e9c6f0529cd7fa539. Latest consensus state: current_term: 2 leader_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:58.893613 11245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:58.894429 11247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:01:58.899477 11247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:01:58.900694 11247 catalog_manager.cc:1259] Loaded cluster ID: 783c9c685ebe49c9a2ad6864a3e32582
I20260927 22:01:58.900794 11247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:01:58.902159 11247 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:01:58.903497 11247 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Loaded TSK: 0
I20260927 22:01:58.904613 11247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:01:58.952184 10688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:01:58.960839 10688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:01:58.966686 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.966249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46034)
0927 22:01:58.966435 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.966518 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 22:01:58.966940 10508 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [term 2 FOLLOWER]: Refusing update from remote peer 9653854c3fde4b5e9c6f0529cd7fa539: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 22:01:58.968129 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.967566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46042)
0927 22:01:58.967780 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.967882 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 22:01:58.968080 11245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 22:01:58.968451 10673 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 -> Peer 082117a1043840558566f1322649c58b (127.3.159.62:33985): Couldn't send request to peer 082117a1043840558566f1322649c58b. Status: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 22:01:58.971761 11235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9653854c3fde4b5e9c6f0529cd7fa539. Latest consensus state: current_term: 2 leader_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:58.972092 11235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:01:58.974226 11236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:58.974663 11236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539 [sys.catalog]: This master's current role is: LEADER
I20260927 22:01:58.974931 11235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "082117a1043840558566f1322649c58b" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33985 } } peers { permanent_uuid: "9ffa375e09fa444187a77ac58b5966ee" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 34347 } attrs { promote: false } } peers { permanent_uuid: "9653854c3fde4b5e9c6f0529cd7fa539" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 34919 } attrs { promote: false } } }
I20260927 22:01:58.975307 11235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ffa375e09fa444187a77ac58b5966ee [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:01:58.996116 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.986567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:37633 (local address 127.0.0.1:41780)
0927 22:01:58.986978 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.986996 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:58.987121 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:58.988373 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:58.988380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:58.988405 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:58.988894 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:58.988905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.990475 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.990480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:58.992821 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:58.992836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.995021 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.995028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:58.995223 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:58.995267 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:01:58.995409 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 22:01:58.995582 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20260927 22:01:58.997407 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.987364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.3.159.1:37633)
0927 22:01:58.987925 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:58.987931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:58.987951 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:01:58.988056 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:58.988106 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:58.988204 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:58.989052 (+ 848us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:58.989182 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.990314 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.993028 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.994702 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.996604 (+ 1902us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:58.996755 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:58.996814 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 22:01:58.996992 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":324,"threads_started":1}
I20260927 22:01:58.997977 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.986344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:33831 (local address 127.0.0.1:43004)
0927 22:01:58.986646 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.986676 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:58.986815 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:58.991225 (+ 4410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:58.991231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:58.991264 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:58.991554 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:58.991568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.993360 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.993366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:58.995005 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:58.995018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.997159 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.997164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:58.997375 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:58.997449 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 22:01:58.997551 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 22:01:58.997712 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":83}
I20260927 22:01:58.998422 10803 tablet_service.cc:1467] Processing CreateTablet for tablet b5a4777376c44f41ab084cfc9d0e6153 (DEFAULT_TABLE table=test-table [id=e532c221b0404039991f5c72b4dac04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:58.998961 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.986411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.3.159.2:33831)
0927 22:01:58.990798 (+ 4387us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:58.990808 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:58.990829 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:01:58.990905 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:58.990911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:58.990988 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:58.991154 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:58.992066 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.993118 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.995222 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.996167 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.996735 (+ 568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:58.996837 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:58.998471 (+ 1634us) server_negotiation.cc:299] Negotiation successful
0927 22:01:58.998639 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4243,"thread_start_us":4111,"threads_started":1}
I20260927 22:01:58.998988 10803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a4777376c44f41ab084cfc9d0e6153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:58.999377 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.988043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.3.159.3:34903)
0927 22:01:58.991014 (+ 2971us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:58.991018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:58.991041 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:01:58.991108 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:58.991112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:58.991233 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:58.992155 (+ 922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:58.992314 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.993231 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.996464 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.997913 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.998075 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:58.998211 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:58.998849 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 22:01:58.999049 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2845,"thread_start_us":2724,"threads_started":1}
I20260927 22:01:58.999617 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:58.987352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:34903 (local address 127.0.0.1:48006)
0927 22:01:58.990155 (+ 2803us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:58.990174 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:58.990293 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:58.991428 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:58.991433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:58.991477 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:58.991996 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:58.992006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.993419 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.993423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:58.996186 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:58.996203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:58.998357 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:58.998363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:58.998555 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:58.998656 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 22:01:58.998760 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:01:58.998898 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2670,"mutex_wait_us":47,"thread_start_us":2363,"threads_started":1}
I20260927 22:01:59.000541 11094 tablet_service.cc:1467] Processing CreateTablet for tablet b5a4777376c44f41ab084cfc9d0e6153 (DEFAULT_TABLE table=test-table [id=e532c221b0404039991f5c72b4dac04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:59.001009 11094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a4777376c44f41ab084cfc9d0e6153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:59.002504 10953 tablet_service.cc:1467] Processing CreateTablet for tablet b5a4777376c44f41ab084cfc9d0e6153 (DEFAULT_TABLE table=test-table [id=e532c221b0404039991f5c72b4dac04a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:01:59.003108 10953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5a4777376c44f41ab084cfc9d0e6153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:01:59.005923 11258 tablet_bootstrap.cc:492] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882: Bootstrap starting.
I20260927 22:01:59.008028 11258 tablet_bootstrap.cc:654] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:59.012924 11258 tablet_bootstrap.cc:492] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882: No bootstrap required, opened a new log
I20260927 22:01:59.013180 11258 ts_tablet_manager.cc:1397] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260927 22:01:59.014564 11258 raft_consensus.cc:348] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } }
I20260927 22:01:59.015026 11258 raft_consensus.cc:374] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:59.015127 11258 raft_consensus.cc:729] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d4d325d1014c8d9daeee0118430882, State: Initialized, Role: FOLLOWER
I20260927 22:01:59.015516 11258 consensus_queue.cc:260] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } }
I20260927 22:01:59.019692 11261 tablet_bootstrap.cc:492] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a: Bootstrap starting.
I20260927 22:01:59.024097 11261 tablet_bootstrap.cc:654] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:59.025058 11258 ts_tablet_manager.cc:1428] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882: Time spent starting tablet: real 0.012s user 0.000s sys 0.010s
I20260927 22:01:59.026054 11260 tablet_bootstrap.cc:492] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962: Bootstrap starting.
I20260927 22:01:59.028435 11260 tablet_bootstrap.cc:654] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962: Neither blocks nor log segments found. Creating new log.
I20260927 22:01:59.031328 10868 heartbeater.cc:503] Master 127.3.159.60:34919 was elected leader, sending a full tablet report...
I20260927 22:01:59.034698 11260 tablet_bootstrap.cc:492] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962: No bootstrap required, opened a new log
I20260927 22:01:59.034932 11260 ts_tablet_manager.cc:1397] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 22:01:59.036032 11260 raft_consensus.cc:348] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } }
I20260927 22:01:59.036363 11260 raft_consensus.cc:374] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:59.036482 11260 raft_consensus.cc:729] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ff1411e4e14671a218145ee48c2962, State: Initialized, Role: FOLLOWER
I20260927 22:01:59.036906 11260 consensus_queue.cc:260] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } }
I20260927 22:01:59.038838 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.029157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.1:40239)
0927 22:01:59.038235 (+ 9078us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.038491 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8826,"thread_start_us":8580,"threads_started":1}
I20260927 22:01:59.041251 11260 ts_tablet_manager.cc:1428] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260927 22:01:59.041548 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.040044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.1:51843)
0927 22:01:59.040253 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.040358 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 22:01:59.042426 11261 tablet_bootstrap.cc:492] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a: No bootstrap required, opened a new log
I20260927 22:01:59.042632 11261 ts_tablet_manager.cc:1397] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20260927 22:01:59.043493 11261 raft_consensus.cc:348] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } }
I20260927 22:01:59.043722 11261 raft_consensus.cc:374] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:01:59.043828 11261 raft_consensus.cc:729] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b8c7e9f16e49fb9a6538dec710976a, State: Initialized, Role: FOLLOWER
I20260927 22:01:59.044103 11261 consensus_queue.cc:260] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } }
I20260927 22:01:59.044937 11261 ts_tablet_manager.cc:1428] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:01:59.046783 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.041835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.2:59521)
0927 22:01:59.046318 (+ 4483us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.046465 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4345,"thread_start_us":4179,"threads_started":1}
I20260927 22:01:59.047529 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.045783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.3:44055)
0927 22:01:59.046813 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.046969 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":616,"threads_started":1}
I20260927 22:01:59.047608 11159 heartbeater.cc:503] Master 127.3.159.60:34919 was elected leader, sending a full tablet report...
I20260927 22:01:59.048283 11018 heartbeater.cc:503] Master 127.3.159.60:34919 was elected leader, sending a full tablet report...
I20260927 22:01:59.049249 11269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.048245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.3:46555)
0927 22:01:59.048918 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.049045 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"threads_started":1}
I20260927 22:01:59.050546 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.047694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.2:44509)
0927 22:01:59.048140 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.050340 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306}
I20260927 22:01:59.166538 11262 raft_consensus.cc:482] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:01:59.166939 11262 raft_consensus.cc:504] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } }
I20260927 22:01:59.169081 11262 leader_election.cc:288] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11ff1411e4e14671a218145ee48c2962 (127.3.159.2:33831), 60b8c7e9f16e49fb9a6538dec710976a (127.3.159.3:34903)
I20260927 22:01:59.176944 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.169725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:34217 (local address 127.3.159.2:33831)
0927 22:01:59.169983 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:59.169991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:59.170015 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:01:59.170272 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:59.170278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:59.170379 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:59.170482 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:59.171320 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.172242 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.174607 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.175850 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.175961 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:59.176041 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:59.176607 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 22:01:59.176719 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 22:01:59.177358 11263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.169491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:33831 (local address 127.3.159.1:34217)
0927 22:01:59.169830 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.169853 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:59.170056 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:59.170543 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:59.170549 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:59.170582 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:59.171020 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:59.171036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.172408 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.172414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:59.174301 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:59.174317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.176123 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.176131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:59.176351 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:59.176398 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 22:01:59.176546 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 22:01:59.176749 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":1,"spinlock_wait_cycles":38272}
I20260927 22:01:59.178479 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.169544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:34903 (local address 127.3.159.1:41377)
0927 22:01:59.170499 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.170546 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 22:01:59.170675 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:01:59.171116 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:01:59.171249 (+ 133us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:01:59.171282 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:01:59.171668 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:59.171680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.173229 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.173233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:59.175357 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:01:59.175378 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.177509 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.177521 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:01:59.177750 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:59.177795 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:01:59.177888 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 22:01:59.178015 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":1,"thread_start_us":635,"threads_started":1}
I20260927 22:01:59.178531 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.169958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:41377 (local address 127.3.159.3:34903)
0927 22:01:59.170226 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:59.170233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:59.170645 (+ 412us) server_negotiation.cc:407] Connection header received
0927 22:01:59.170824 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:59.170829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:59.170915 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:59.171061 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:01:59.171884 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.173011 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.175638 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.176980 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.177148 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:59.177300 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:59.177968 (+ 668us) server_negotiation.cc:299] Negotiation successful
0927 22:01:59.178143 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:01:59.178627 10973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a4777376c44f41ab084cfc9d0e6153" candidate_uuid: "00d4d325d1014c8d9daeee0118430882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ff1411e4e14671a218145ee48c2962" is_pre_election: true
I20260927 22:01:59.179164 10973 raft_consensus.cc:2454] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00d4d325d1014c8d9daeee0118430882 in term 0.
I20260927 22:01:59.179385 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a4777376c44f41ab084cfc9d0e6153" candidate_uuid: "00d4d325d1014c8d9daeee0118430882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b8c7e9f16e49fb9a6538dec710976a" is_pre_election: true
I20260927 22:01:59.179677 11114 raft_consensus.cc:2454] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00d4d325d1014c8d9daeee0118430882 in term 0.
I20260927 22:01:59.179857 10759 leader_election.cc:302] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d4d325d1014c8d9daeee0118430882, 11ff1411e4e14671a218145ee48c2962; no voters:
I20260927 22:01:59.180500 11262 raft_consensus.cc:2781] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:01:59.180644 11262 raft_consensus.cc:482] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:01:59.180763 11262 raft_consensus.cc:3037] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:59.183286 11262 raft_consensus.cc:504] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } }
I20260927 22:01:59.184093 11262 leader_election.cc:288] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [CANDIDATE]: Term 1 election: Requested vote from peers 11ff1411e4e14671a218145ee48c2962 (127.3.159.2:33831), 60b8c7e9f16e49fb9a6538dec710976a (127.3.159.3:34903)
I20260927 22:01:59.184860 10973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a4777376c44f41ab084cfc9d0e6153" candidate_uuid: "00d4d325d1014c8d9daeee0118430882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ff1411e4e14671a218145ee48c2962"
I20260927 22:01:59.185043 11114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5a4777376c44f41ab084cfc9d0e6153" candidate_uuid: "00d4d325d1014c8d9daeee0118430882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b8c7e9f16e49fb9a6538dec710976a"
I20260927 22:01:59.185156 10973 raft_consensus.cc:3037] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:59.185277 11114 raft_consensus.cc:3037] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:01:59.189122 11114 raft_consensus.cc:2454] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d4d325d1014c8d9daeee0118430882 in term 1.
I20260927 22:01:59.189122 10973 raft_consensus.cc:2454] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d4d325d1014c8d9daeee0118430882 in term 1.
I20260927 22:01:59.189772 10759 leader_election.cc:302] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d4d325d1014c8d9daeee0118430882, 11ff1411e4e14671a218145ee48c2962; no voters:
I20260927 22:01:59.190274 11262 raft_consensus.cc:2781] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:01:59.190881 11262 raft_consensus.cc:686] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 1 LEADER]: Becoming Leader. State: Replica: 00d4d325d1014c8d9daeee0118430882, State: Running, Role: LEADER
I20260927 22:01:59.191459 11262 consensus_queue.cc:237] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } }
I20260927 22:01:59.192250 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.191749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.1:47685)
0927 22:01:59.191904 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.192106 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 22:01:59.196573 10688 catalog_manager.cc:5495] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00d4d325d1014c8d9daeee0118430882 (127.3.159.1). New cstate: current_term: 1 leader_uuid: "00d4d325d1014c8d9daeee0118430882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00d4d325d1014c8d9daeee0118430882" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 37633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 } health_report { overall_health: UNKNOWN } } }
I20260927 22:01:59.250265 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.243899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.3.159.1:37633)
0927 22:01:59.244175 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:59.244181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:59.244207 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:01:59.244442 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:59.244446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:59.244569 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:59.244782 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:01:59.245199 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.246078 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.247531 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.248030 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.248186 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:59.248662 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:01:59.249522 (+ 860us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:01:59.249698 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:59.249809 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 22:01:59.249912 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 22:01:59.255980 11114 raft_consensus.cc:1261] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a [term 1 FOLLOWER]: Refusing update from remote peer 00d4d325d1014c8d9daeee0118430882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:59.256387 10973 raft_consensus.cc:1261] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962 [term 1 FOLLOWER]: Refusing update from remote peer 00d4d325d1014c8d9daeee0118430882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:01:59.256889 11272 consensus_queue.cc:1035] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b8c7e9f16e49fb9a6538dec710976a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:59.257433 11262 consensus_queue.cc:1035] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11ff1411e4e14671a218145ee48c2962" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:01:59.260747 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.259795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.3:42491)
0927 22:01:59.260117 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.260323 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 22:01:59.260748 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.259925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.2:32849)
0927 22:01:59.260268 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.260454 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260927 22:01:59.266922 11275 mvcc.cc:205] Tried to move back new op lower bound from 7334078542863396864 to 7334078542613413888. Current Snapshot: MvccSnapshot[applied={T|T < 7334078542863396864}]
I20260927 22:01:59.269930 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.269339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.1:35129)
0927 22:01:59.269592 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.269711 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260927 22:01:59.270766 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.270278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.2:45185)
0927 22:01:59.270475 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.270573 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:01:59.273590 11276 mvcc.cc:205] Tried to move back new op lower bound from 7334078542863396864 to 7334078542613413888. Current Snapshot: MvccSnapshot[applied={T|T < 7334078542863396864}]
I20260927 22:01:59.279874 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.279096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.1:50385)
0927 22:01:59.279488 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.279620 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254}
I20260927 22:01:59.288659 11270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.287076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.3.159.3:39855)
0927 22:01:59.287989 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.288175 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":631}
I20260927 22:01:59.303507 10688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:59.303920 10688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:01:59.312247 10688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9653854c3fde4b5e9c6f0529cd7fa539: Sending DeleteTablet for 3 replicas of tablet b5a4777376c44f41ab084cfc9d0e6153
I20260927 22:01:59.313354 10953 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a4777376c44f41ab084cfc9d0e6153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:59 UTC) from {username='oryx', principal='oryx/127.3.159.60@KRBTEST.COM'} at 127.0.0.1:43004
I20260927 22:01:59.313808 11094 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a4777376c44f41ab084cfc9d0e6153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:59 UTC) from {username='oryx', principal='oryx/127.3.159.60@KRBTEST.COM'} at 127.0.0.1:48006
I20260927 22:01:59.313885 10803 tablet_service.cc:1514] Processing DeleteTablet for tablet b5a4777376c44f41ab084cfc9d0e6153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:01:59 UTC) from {username='oryx', principal='oryx/127.3.159.60@KRBTEST.COM'} at 127.0.0.1:41780
I20260927 22:01:59.320048 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 10670
I20260927 22:01:59.332314 11285 tablet_replica.cc:330] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962: stopping tablet replica
I20260927 22:01:59.332731 11285 raft_consensus.cc:2229] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:59.333232 11285 raft_consensus.cc:2258] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:59.334858 11285 ts_tablet_manager.cc:1905] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:59.342214 11286 tablet_replica.cc:330] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a: stopping tablet replica
I20260927 22:01:59.342491 11286 raft_consensus.cc:2229] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:01:59.342824 11286 raft_consensus.cc:2258] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:59.343636 11287 tablet_replica.cc:330] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882: stopping tablet replica
I20260927 22:01:59.343914 11285 ts_tablet_manager.cc:1918] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:59.344089 11285 log.cc:1196] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/b5a4777376c44f41ab084cfc9d0e6153
I20260927 22:01:59.343971 11287 raft_consensus.cc:2229] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:01:59.344571 11285 ts_tablet_manager.cc:1939] T b5a4777376c44f41ab084cfc9d0e6153 P 11ff1411e4e14671a218145ee48c2962: Deleting consensus metadata
I20260927 22:01:59.345021 11287 raft_consensus.cc:2258] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:01:59.345888 10676 catalog_manager.cc:4841] TS 11ff1411e4e14671a218145ee48c2962 (127.3.159.2:33831): tablet b5a4777376c44f41ab084cfc9d0e6153 (table test-table [id=e532c221b0404039991f5c72b4dac04a]) successfully deleted
I20260927 22:01:59.346848 11286 ts_tablet_manager.cc:1905] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:59.346910 11287 ts_tablet_manager.cc:1905] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:01:59.355994 11286 ts_tablet_manager.cc:1918] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:59.356168 11286 log.cc:1196] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/b5a4777376c44f41ab084cfc9d0e6153
I20260927 22:01:59.356451 11287 ts_tablet_manager.cc:1918] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:01:59.356717 11286 ts_tablet_manager.cc:1939] T b5a4777376c44f41ab084cfc9d0e6153 P 60b8c7e9f16e49fb9a6538dec710976a: Deleting consensus metadata
I20260927 22:01:59.356735 11287 log.cc:1196] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/b5a4777376c44f41ab084cfc9d0e6153
I20260927 22:01:59.357534 11287 ts_tablet_manager.cc:1939] T b5a4777376c44f41ab084cfc9d0e6153 P 00d4d325d1014c8d9daeee0118430882: Deleting consensus metadata
I20260927 22:01:59.357954 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.321984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.3.159.1:37633)
0927 22:01:59.322197 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:59.322204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:59.322865 (+ 661us) server_negotiation.cc:407] Connection header received
0927 22:01:59.322960 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:59.322968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:59.323087 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:59.323228 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:59.325139 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.327937 (+ 2798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.337911 (+ 9974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.338696 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.338820 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:59.346799 (+ 7979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:59.346826 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:59.346856 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:59.346934 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:59.352338 (+ 5404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:59.353638 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:59.353651 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:59.353655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:59.353900 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:59.354529 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:59.354534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:59.354538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:59.355245 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:59.355639 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:59.356423 (+ 784us) server_negotiation.cc:299] Negotiation successful
0927 22:01:59.356929 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:01:59.358222 10675 catalog_manager.cc:4841] TS 60b8c7e9f16e49fb9a6538dec710976a (127.3.159.3:34903): tablet b5a4777376c44f41ab084cfc9d0e6153 (table test-table [id=e532c221b0404039991f5c72b4dac04a]) successfully deleted
I20260927 22:01:59.359434 10863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:59.369279 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.368854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46058)
0927 22:01:59.369028 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.369132 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 22:01:59.370357 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.370013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46060)
0927 22:01:59.370162 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.370236 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 22:01:59.492540 10673 catalog_manager.cc:4841] TS 00d4d325d1014c8d9daeee0118430882 (127.3.159.1:37633): tablet b5a4777376c44f41ab084cfc9d0e6153 (table test-table [id=e532c221b0404039991f5c72b4dac04a]) successfully deleted
I20260927 22:01:59.494326 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 10670
I20260927 22:01:59.516425 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 10880
I20260927 22:01:59.533937 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.517532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.3.159.2:33831)
0927 22:01:59.517817 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:59.517826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:59.517850 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:01:59.517925 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:59.517930 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:59.518005 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:59.518116 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:59.519590 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.520827 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.522425 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.522874 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.523004 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:59.526099 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:59.526119 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:59.526142 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:59.526199 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:59.530257 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:59.531316 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:59.531326 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:59.531329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:59.531517 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:59.532098 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:59.532108 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:59.532111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:59.532598 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:59.532950 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:59.533260 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 22:01:59.533571 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 22:01:59.534761 11013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:59.662129 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 10880
I20260927 22:01:59.689015 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 11031
I20260927 22:01:59.705245 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.690251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48012 (local address 127.3.159.3:34903)
0927 22:01:59.691039 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0927 22:01:59.691044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:01:59.691062 (+ 18us) server_negotiation.cc:407] Connection header received
0927 22:01:59.691122 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:01:59.691126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:01:59.691228 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:01:59.691372 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:01:59.692851 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.693704 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.694999 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:01:59.695395 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:01:59.695535 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:01:59.699145 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:01:59.699166 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:01:59.699183 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:01:59.699222 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:01:59.702353 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:59.703125 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:59.703144 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:59.703146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:59.703265 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:01:59.703652 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:01:59.703658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:01:59.703662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:01:59.704046 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:01:59.704343 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 22:01:59.704648 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 22:01:59.704873 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":337,"threads_started":1}
I20260927 22:01:59.706293 11154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:59.808003 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 11031
I20260927 22:01:59.830348 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 10362
I20260927 22:01:59.831779 10529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:01:59.965916 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.965005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46066)
0927 22:01:59.965606 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.965728 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":315,"threads_started":1}
I20260927 22:01:59.967654 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:01:59.967247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:33985 (local address 127.0.0.1:46080)
0927 22:01:59.967424 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 22:01:59.967530 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:33985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:01:59.972863 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 10362
I20260927 22:01:59.999480 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 10465
I20260927 22:02:00.000802 10719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:00.142632 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 10465
2026-09-27T22:02:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11246 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:02:00 dist-test-slave-sg4j krb5kdc[11299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:02:00 dist-test-slave-sg4j krb5kdc[11299](info): set up 2 sockets
Sep 27 22:02:00 dist-test-slave-sg4j krb5kdc[11299](info): commencing operation
krb5kdc: starting...
W20260927 22:02:01.354943 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:02:01 dist-test-slave-sg4j krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:02:01Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.62.keytab.
Entry for principal oryx/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.62.keytab.
I20260927 22:02:01.622545 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:39931
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.62.keytab
--principal=oryx/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:39931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:02:02.053889 11312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:02.054311 11312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:02.054426 11312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:02.064198 11312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:02:02.064392 11312 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 22:02:02.064440 11312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:02.064502 11312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:02.064559 11312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:02:02.064634 11312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:02:02.077189 11312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:39931
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.62.keytab
--principal=oryx/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:39931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:02.079198 11312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:02.081061 11312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:02.097239 11320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:02.097086 11317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:02.097241 11318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:02.099350 11312 server_base.cc:1034] running on GCE node
I20260927 22:02:02.100378 11312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:02.102111 11312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:02.103394 11312 hybrid_clock.cc:648] HybridClock initialized: now 1790546522103304 us; error 76 us; skew 500 ppm
I20260927 22:02:02.103814 11312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:02 dist-test-slave-sg4j krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546522, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:02.111150 11312 init.cc:377] Logged in from keytab as oryx/127.3.159.62@KRBTEST.COM (short username oryx)
I20260927 22:02:02.113088 11312 webserver.cc:460] Webserver started at http://127.3.159.62:33707/ using document root <none> and password file <none>
I20260927 22:02:02.114024 11312 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:02.114162 11312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:02.114521 11312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:02.117928 11312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "1240caa533354dd68190d446cd56c853"
format_stamp: "Formatted at 2026-09-27 22:02:02 on dist-test-slave-sg4j"
server_key: "bd926d7e27ab94f1db3f5ae8ad81df7d"
server_key_iv: "d3fc2c5b36233bebae3fd80486393882"
server_key_version: "encryptionkey@0"
I20260927 22:02:02.118817 11312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "1240caa533354dd68190d446cd56c853"
format_stamp: "Formatted at 2026-09-27 22:02:02 on dist-test-slave-sg4j"
server_key: "bd926d7e27ab94f1db3f5ae8ad81df7d"
server_key_iv: "d3fc2c5b36233bebae3fd80486393882"
server_key_version: "encryptionkey@0"
I20260927 22:02:02.125018 11312 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 22:02:02.129930 11327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:02.132181 11312 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:02:02.132452 11312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "1240caa533354dd68190d446cd56c853"
format_stamp: "Formatted at 2026-09-27 22:02:02 on dist-test-slave-sg4j"
server_key: "bd926d7e27ab94f1db3f5ae8ad81df7d"
server_key_iv: "d3fc2c5b36233bebae3fd80486393882"
server_key_version: "encryptionkey@0"
I20260927 22:02:02.132735 11312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:02.150372 11312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:02.155283 11312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:02.155598 11312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:02.182164 11312 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:39931
I20260927 22:02:02.182180 11388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:39931 every 8 connection(s)
I20260927 22:02:02.183779 11312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:02:02.188959 11389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:02.192611 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 11312
I20260927 22:02:02.192833 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:02:02.193646 3708 external_mini_cluster.cc:1428] Setting key 97b847540d81bedbf11570c287abf557
I20260927 22:02:02.205077 11389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853: Bootstrap starting.
I20260927 22:02:02.209734 11389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:02.210955 11389 log.cc:824] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:02:02 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:02:02.215083 11389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853: No bootstrap required, opened a new log
I20260927 22:02:02.223106 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:02.196223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.3.159.62:39931)
0927 22:02:02.197290 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:02.197311 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:02.197413 (+ 102us) server_negotiation.cc:407] Connection header received
0927 22:02:02.199957 (+ 2544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:02.200001 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:02.200771 (+ 770us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:02.201429 (+ 658us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:02.202950 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:02.205121 (+ 2171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:02.206770 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:02.208090 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:02.208327 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:02.213513 (+ 5186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:02.213526 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:02.213548 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:02.213602 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:02.217487 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:02.218716 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:02.218725 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:02.218733 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:02.218972 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:02.219557 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:02.219596 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:02.219598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:02.220339 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:02.220634 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:02.221234 (+ 600us) server_negotiation.cc:299] Negotiation successful
0927 22:02:02.221850 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":466,"threads_started":1}
I20260927 22:02:02.225209 11389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1240caa533354dd68190d446cd56c853" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39931 } }
I20260927 22:02:02.225770 11389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:02.225899 11389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1240caa533354dd68190d446cd56c853, State: Initialized, Role: FOLLOWER
I20260927 22:02:02.226637 11389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1240caa533354dd68190d446cd56c853" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39931 } }
I20260927 22:02:02.226871 11389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:02:02.226979 11389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:02:02.227098 11389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:02.229598 11389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1240caa533354dd68190d446cd56c853" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39931 } }
I20260927 22:02:02.230180 11389 leader_election.cc:302] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1240caa533354dd68190d446cd56c853; no voters:
I20260927 22:02:02.230922 11389 leader_election.cc:288] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:02:02.231278 11394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:02:02.232846 11394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [term 1 LEADER]: Becoming Leader. State: Replica: 1240caa533354dd68190d446cd56c853, State: Running, Role: LEADER
I20260927 22:02:02.233781 11389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:02:02.233650 11394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1240caa533354dd68190d446cd56c853" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39931 } }
I20260927 22:02:02.237113 11396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1240caa533354dd68190d446cd56c853. Latest consensus state: current_term: 1 leader_uuid: "1240caa533354dd68190d446cd56c853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1240caa533354dd68190d446cd56c853" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39931 } } }
I20260927 22:02:02.237725 11396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:02.238304 11395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1240caa533354dd68190d446cd56c853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1240caa533354dd68190d446cd56c853" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39931 } } }
I20260927 22:02:02.238664 11395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853 [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:02.246511 11401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:02:02.252135 11401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:02:02.262436 11401 catalog_manager.cc:1347] Generated new cluster ID: 6cf7cdbfd8b74970916dd27dd95aeb4e
I20260927 22:02:02.262640 11401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:02:02.317741 11401 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:02:02.319311 11401 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:02:02.329332 11401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853: Generated new TSK 0
I20260927 22:02:02.330405 11401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab.
Entry for principal oryx/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab.
I20260927 22:02:02.410200 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:39931
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab
--principal=oryx/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:02:02.789099 11415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:02.789528 11415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:02.789641 11415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:02.797511 11415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:02.797652 11415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:02.797740 11415 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:02:02.797827 11415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:02:02.809448 11415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab
--principal=oryx/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.159.62:39931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:02.810813 11415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:02.812271 11415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:02.827885 11421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:02.828008 11423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:02.827913 11420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:02.829483 11415 server_base.cc:1034] running on GCE node
I20260927 22:02:02.830219 11415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:02.831211 11415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:02.832512 11415 hybrid_clock.cc:648] HybridClock initialized: now 1790546522832413 us; error 62 us; skew 500 ppm
I20260927 22:02:02.832909 11415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:02 dist-test-slave-sg4j krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546522, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:02.837080 11415 init.cc:377] Logged in from keytab as oryx/127.3.159.1@KRBTEST.COM (short username oryx)
I20260927 22:02:02.839079 11415 webserver.cc:460] Webserver started at http://127.3.159.1:40089/ using document root <none> and password file <none>
I20260927 22:02:02.839872 11415 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:02.840021 11415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:02.840384 11415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:02.843724 11415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "cc9a8f4a2b7f47afb5790bba81fec074"
format_stamp: "Formatted at 2026-09-27 22:02:02 on dist-test-slave-sg4j"
server_key: "c433fdc32f44eaa9f480ea027d773593"
server_key_iv: "ef9fbabac77f79b1fe8b06eea42a9a73"
server_key_version: "encryptionkey@0"
I20260927 22:02:02.844689 11415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "cc9a8f4a2b7f47afb5790bba81fec074"
format_stamp: "Formatted at 2026-09-27 22:02:02 on dist-test-slave-sg4j"
server_key: "c433fdc32f44eaa9f480ea027d773593"
server_key_iv: "ef9fbabac77f79b1fe8b06eea42a9a73"
server_key_version: "encryptionkey@0"
I20260927 22:02:02.850849 11415 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 22:02:02.855365 11430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:02.857704 11415 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:02:02.858018 11415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "cc9a8f4a2b7f47afb5790bba81fec074"
format_stamp: "Formatted at 2026-09-27 22:02:02 on dist-test-slave-sg4j"
server_key: "c433fdc32f44eaa9f480ea027d773593"
server_key_iv: "ef9fbabac77f79b1fe8b06eea42a9a73"
server_key_version: "encryptionkey@0"
I20260927 22:02:02.858434 11415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:02.884871 11415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:02.890704 11415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:02.891088 11415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:02.894866 11415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:02.894977 11415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:02.895228 11415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:02.895303 11415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:02:02 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546522, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:02:02.933290 11439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:02.896115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:44895 (local address 127.3.159.62:39931)
0927 22:02:02.896726 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:02.896734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:02.898877 (+ 2143us) server_negotiation.cc:407] Connection header received
0927 22:02:02.902330 (+ 3453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:02.902339 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:02.903216 (+ 877us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:02.903395 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:02.907134 (+ 3739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:02.909164 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:02.912958 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:02.913714 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:02.913888 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:02.921899 (+ 8011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:02.921915 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:02.921924 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:02.922000 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:02.926966 (+ 4966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:02.928985 (+ 2019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:02.929000 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:02.929008 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:02.929259 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:02.930193 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:02.930203 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:02.930211 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:02.930789 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:02.931026 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:02.932261 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0927 22:02:02.932674 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":289,"threads_started":1}
I20260927 22:02:02.936462 11440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:02.897229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:39931 (local address 127.3.159.1:44895)
0927 22:02:02.898592 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:02.898662 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:02.901711 (+ 3049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:02.904055 (+ 2344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:02.904075 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:02.905234 (+ 1159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:02.906714 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:02.906756 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:02.909617 (+ 2861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:02.909628 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:02.912617 (+ 2989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:02.912649 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:02.914444 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:02.914456 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:02.915209 (+ 753us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:02.916715 (+ 1506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:02.916770 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:02.921284 (+ 4514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:02.927265 (+ 5981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:02.927285 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:02.927306 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:02.928680 (+ 1374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:02.929496 (+ 816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:02.929504 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:02.929511 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:02.929958 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:02.931005 (+ 1047us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:02.931017 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:02.931589 (+ 572us) client_negotiation.cc:769] Sending connection context
0927 22:02:02.932055 (+ 466us) client_negotiation.cc:240] Negotiation successful
0927 22:02:02.932830 (+ 775us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"thread_start_us":526,"threads_started":1}
I20260927 22:02:02.963574 11415 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:34009
I20260927 22:02:02.963625 11550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:34009 every 8 connection(s)
I20260927 22:02:02.965507 11415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:02:02.966766 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 11415
I20260927 22:02:02.967254 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:02:02.968116 3708 external_mini_cluster.cc:1428] Setting key ee19d7e9056ec083deaac028575d1fb9
I20260927 22:02:02.969831 11551 heartbeater.cc:347] Connected to a master server at 127.3.159.62:39931
I20260927 22:02:02.970999 11551 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:02.971678 11439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:02.950509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:42357 (local address 127.3.159.62:39931)
0927 22:02:02.950739 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:02.950745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:02.951679 (+ 934us) server_negotiation.cc:407] Connection header received
0927 22:02:02.951974 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:02.951979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:02.952063 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:02.952242 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:02.954192 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:02.955191 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:02.958378 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:02.959142 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:02.959326 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:02.962628 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:02.962636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:02.962643 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:02.962703 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:02.967360 (+ 4657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:02.968908 (+ 1548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:02.968915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:02.968919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:02.969133 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:02.969694 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:02.969701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:02.969704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:02.970104 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:02.970277 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:02.970937 (+ 660us) server_negotiation.cc:299] Negotiation successful
0927 22:02:02.971271 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 22:02:02.971807 11526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:02.950575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:39931 (local address 127.3.159.1:42357)
0927 22:02:02.951456 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:02.951494 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:02.951755 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:02.952283 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:02.952289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:02.953276 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:02.953983 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:02.953998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:02.955482 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:02.955499 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:02.958109 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:02.958146 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:02.959477 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:02.959487 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:02.959829 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:02.960803 (+ 974us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:02.960828 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:02.962239 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:02.967617 (+ 5378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:02.967624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:02.967631 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:02.968659 (+ 1028us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:02.969303 (+ 644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:02.969308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:02.969313 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:02.969519 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:02.970256 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:02.970262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:02.970550 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 22:02:02.970794 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 22:02:02.971256 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":457,"threads_started":1}
I20260927 22:02:02.971946 11551 heartbeater.cc:511] Master 127.3.159.62:39931 requested a full tablet report, sending...
I20260927 22:02:02.974670 11343 ts_manager.cc:194] Registered new tserver with Master: cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1:34009)
I20260927 22:02:02.976548 11343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:44895
WARNING: no policy specified for oryx/127.3.159.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab.
Entry for principal oryx/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab.
I20260927 22:02:03.030246 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:39931
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab
--principal=oryx/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:02:03.431120 11555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:03.431550 11555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:03.431738 11555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:03.440747 11555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:03.440905 11555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:03.441071 11555 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:02:03.441175 11555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:02:03.453356 11555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab
--principal=oryx/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.159.62:39931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:03.455180 11555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:03.456858 11555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:03.473461 11560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:03.473745 11563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:03.473428 11561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:03.474932 11555 server_base.cc:1034] running on GCE node
I20260927 22:02:03.475780 11555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:03.477030 11555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:03.478284 11555 hybrid_clock.cc:648] HybridClock initialized: now 1790546523478163 us; error 83 us; skew 500 ppm
I20260927 22:02:03.478669 11555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:03 dist-test-slave-sg4j krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:03.482764 11555 init.cc:377] Logged in from keytab as oryx/127.3.159.2@KRBTEST.COM (short username oryx)
I20260927 22:02:03.484743 11555 webserver.cc:460] Webserver started at http://127.3.159.2:36273/ using document root <none> and password file <none>
I20260927 22:02:03.485759 11555 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:03.485867 11555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:03.486255 11555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:03.489985 11555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "9f6b55eeb6d74baa97e4f23497e03b01"
format_stamp: "Formatted at 2026-09-27 22:02:03 on dist-test-slave-sg4j"
server_key: "4c59bb3ae445a8f10b8d05681421d041"
server_key_iv: "893a261eae94a064e8b78f90411903c5"
server_key_version: "encryptionkey@0"
I20260927 22:02:03.490955 11555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "9f6b55eeb6d74baa97e4f23497e03b01"
format_stamp: "Formatted at 2026-09-27 22:02:03 on dist-test-slave-sg4j"
server_key: "4c59bb3ae445a8f10b8d05681421d041"
server_key_iv: "893a261eae94a064e8b78f90411903c5"
server_key_version: "encryptionkey@0"
I20260927 22:02:03.497936 11555 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260927 22:02:03.502750 11570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:03.504812 11555 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:02:03.505046 11555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "9f6b55eeb6d74baa97e4f23497e03b01"
format_stamp: "Formatted at 2026-09-27 22:02:03 on dist-test-slave-sg4j"
server_key: "4c59bb3ae445a8f10b8d05681421d041"
server_key_iv: "893a261eae94a064e8b78f90411903c5"
server_key_version: "encryptionkey@0"
I20260927 22:02:03.505309 11555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:03.521633 11555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:03.527228 11555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:03.527598 11555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:03.531097 11555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:03.531201 11555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:03.531454 11555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:03.531523 11555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:02:03 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546523, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:02:03.560825 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:03.531960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:52169 (local address 127.3.159.62:39931)
0927 22:02:03.532569 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:03.532577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:03.534416 (+ 1839us) server_negotiation.cc:407] Connection header received
0927 22:02:03.538591 (+ 4175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:03.538599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:03.538718 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:03.539166 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:03.542089 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:03.542971 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:03.545101 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:03.545616 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:03.545754 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:03.552299 (+ 6545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:03.552306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:03.552309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:03.552348 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:03.556264 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:03.557198 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:03.557204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:03.557210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:03.557336 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:03.557892 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:03.557900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:03.557903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:03.558191 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:03.558347 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:03.559645 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0927 22:02:03.560123 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":319,"threads_started":1}
I20260927 22:02:03.561266 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:03.532690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:39931 (local address 127.3.159.2:52169)
0927 22:02:03.534109 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:03.534210 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:03.537997 (+ 3787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:03.539517 (+ 1520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:03.539530 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:03.540644 (+ 1114us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:03.541819 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:03.541845 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:03.543165 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:03.543173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:03.544929 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:03.544943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:03.545920 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:03.545930 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:03.546142 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:03.547107 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:03.547132 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:03.551976 (+ 4844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:03.556466 (+ 4490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:03.556477 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:03.556501 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:03.557037 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:03.557489 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:03.557500 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:03.557504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:03.557757 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:03.558338 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:03.558348 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:03.558972 (+ 624us) client_negotiation.cc:769] Sending connection context
0927 22:02:03.559317 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 22:02:03.559886 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":617,"threads_started":1}
I20260927 22:02:03.589689 11622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:03.568402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:39931 (local address 127.3.159.2:40309)
0927 22:02:03.570310 (+ 1908us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:03.570346 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:03.570673 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:03.571218 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:03.571228 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:03.572079 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:03.572633 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:03.572659 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:03.573959 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:03.573966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:03.576517 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:03.576559 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:03.577891 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:03.577903 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:03.578233 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:03.579303 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:03.579342 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:03.580880 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:03.585650 (+ 4770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:03.585657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:03.585664 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:03.586352 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:03.586865 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:03.586871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:03.586877 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:03.587085 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:03.587876 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:03.587882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:03.588346 (+ 464us) client_negotiation.cc:769] Sending connection context
0927 22:02:03.588546 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 22:02:03.588978 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1634,"thread_start_us":1377,"threads_started":1}
I20260927 22:02:03.590260 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:03.568438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:40309 (local address 127.3.159.62:39931)
0927 22:02:03.568798 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:03.568810 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:03.570541 (+ 1731us) server_negotiation.cc:407] Connection header received
0927 22:02:03.570904 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:03.570911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:03.571024 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:03.571118 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:03.572906 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:03.573780 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:03.576906 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:03.577537 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:03.577744 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:03.581237 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:03.581244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:03.581248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:03.581292 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:03.585321 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:03.586513 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:03.586521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:03.586525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:03.586673 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:03.587238 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:03.587244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:03.587248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:03.587757 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:03.587898 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:03.589665 (+ 1767us) server_negotiation.cc:299] Negotiation successful
0927 22:02:03.589920 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260927 22:02:03.607213 11555 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:33197
I20260927 22:02:03.607245 11690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:33197 every 8 connection(s)
I20260927 22:02:03.608846 11555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:02:03.610334 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 11555
I20260927 22:02:03.610569 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:02:03.611093 3708 external_mini_cluster.cc:1428] Setting key 66739110ce6f82db21a72f423e0bfa6b
I20260927 22:02:03.612039 11691 heartbeater.cc:347] Connected to a master server at 127.3.159.62:39931
I20260927 22:02:03.612357 11691 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:03.613371 11691 heartbeater.cc:511] Master 127.3.159.62:39931 requested a full tablet report, sending...
I20260927 22:02:03.615244 11343 ts_manager.cc:194] Registered new tserver with Master: 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2:33197)
I20260927 22:02:03.616207 11343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:52169
WARNING: no policy specified for oryx/127.3.159.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab.
Entry for principal oryx/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab.
I20260927 22:02:03.670311 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:39931
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab
--principal=oryx/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:45021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:02:03.980046 11551 heartbeater.cc:503] Master 127.3.159.62:39931 was elected leader, sending a full tablet report...
W20260927 22:02:04.098618 11695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:04.098991 11695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:04.099100 11695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:04.107558 11695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:04.107738 11695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:04.107838 11695 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 22:02:04.107930 11695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:02:04.119393 11695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:45021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab
--principal=oryx/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.159.62:39931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:04.121058 11695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:04.122715 11695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:04.138684 11703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:04.138747 11701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:04.138940 11700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:04.140437 11695 server_base.cc:1034] running on GCE node
I20260927 22:02:04.141358 11695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:04.142879 11695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:04.144399 11695 hybrid_clock.cc:648] HybridClock initialized: now 1790546524144254 us; error 68 us; skew 500 ppm
I20260927 22:02:04.144937 11695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:04 dist-test-slave-sg4j krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:04.149614 11695 init.cc:377] Logged in from keytab as oryx/127.3.159.3@KRBTEST.COM (short username oryx)
I20260927 22:02:04.151747 11695 webserver.cc:460] Webserver started at http://127.3.159.3:33685/ using document root <none> and password file <none>
I20260927 22:02:04.152787 11695 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:04.152956 11695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:04.153488 11695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:04.157536 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "63a16926ad454239b4914139056e2a2a"
format_stamp: "Formatted at 2026-09-27 22:02:04 on dist-test-slave-sg4j"
server_key: "5343717054520356b21d99bb92ff7091"
server_key_iv: "ad2440c6f07f0cec87ff539164668e32"
server_key_version: "encryptionkey@0"
I20260927 22:02:04.158600 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "63a16926ad454239b4914139056e2a2a"
format_stamp: "Formatted at 2026-09-27 22:02:04 on dist-test-slave-sg4j"
server_key: "5343717054520356b21d99bb92ff7091"
server_key_iv: "ad2440c6f07f0cec87ff539164668e32"
server_key_version: "encryptionkey@0"
I20260927 22:02:04.166342 11695 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260927 22:02:04.171025 11710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:04.173202 11695 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260927 22:02:04.173522 11695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "63a16926ad454239b4914139056e2a2a"
format_stamp: "Formatted at 2026-09-27 22:02:04 on dist-test-slave-sg4j"
server_key: "5343717054520356b21d99bb92ff7091"
server_key_iv: "ad2440c6f07f0cec87ff539164668e32"
server_key_version: "encryptionkey@0"
I20260927 22:02:04.173844 11695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:04.210438 11695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:04.215274 11695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:04.215600 11695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:04.218904 11695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:04.219035 11695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:04.219234 11695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:04.219317 11695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:02:04 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546524, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:02:04.252067 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.220404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:55761 (local address 127.3.159.62:39931)
0927 22:02:04.221109 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:04.221116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:04.222773 (+ 1657us) server_negotiation.cc:407] Connection header received
0927 22:02:04.226284 (+ 3511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:04.226292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:04.226422 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:04.226584 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:04.231142 (+ 4558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.232442 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.235597 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.236221 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.236390 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.242940 (+ 6550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:04.242947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:04.242951 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:04.242998 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:04.246796 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:04.248384 (+ 1588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:04.248394 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:04.248399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:04.248581 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:04.249461 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:04.249466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:04.249469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:04.249956 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:04.250147 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:04.251513 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0927 22:02:04.251740 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":376,"threads_started":1}
I20260927 22:02:04.254415 11720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.221245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:39931 (local address 127.3.159.3:55761)
0927 22:02:04.222497 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:04.222567 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:04.225711 (+ 3144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:04.227087 (+ 1376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:04.227104 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:04.228435 (+ 1331us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:04.229912 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.229956 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.232693 (+ 2737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.232699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.235375 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.235400 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.236555 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.236566 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.236875 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.238036 (+ 1161us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:04.238128 (+ 92us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:04.242564 (+ 4436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:04.247078 (+ 4514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:04.247094 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:04.247120 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:04.248150 (+ 1030us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:04.248738 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:04.248748 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:04.248752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:04.249284 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:04.250160 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:04.250178 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:04.250922 (+ 744us) client_negotiation.cc:769] Sending connection context
0927 22:02:04.251409 (+ 487us) client_negotiation.cc:240] Negotiation successful
0927 22:02:04.252231 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":515,"threads_started":1}
I20260927 22:02:04.282629 11695 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:44265
I20260927 22:02:04.282749 11830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:44265 every 8 connection(s)
I20260927 22:02:04.284238 11695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:02:04.287658 11831 heartbeater.cc:347] Connected to a master server at 127.3.159.62:39931
I20260927 22:02:04.288033 11831 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:04.288945 11831 heartbeater.cc:511] Master 127.3.159.62:39931 requested a full tablet report, sending...
I20260927 22:02:04.290482 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.270097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:49083 (local address 127.3.159.62:39931)
0927 22:02:04.270365 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:04.270375 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:04.271380 (+ 1005us) server_negotiation.cc:407] Connection header received
0927 22:02:04.271704 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:04.271711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:04.271827 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:04.271959 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:04.274319 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.275651 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.278934 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.279681 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.279795 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.283353 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:04.283361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:04.283366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:04.283419 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:04.287246 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:04.288213 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:04.288222 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:04.288231 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:04.288388 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:04.288807 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:04.288811 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:04.288813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:04.289179 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:04.289357 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:04.289815 (+ 458us) server_negotiation.cc:299] Negotiation successful
0927 22:02:04.290073 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:02:04.290525 11806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.270159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:39931 (local address 127.3.159.3:49083)
0927 22:02:04.271211 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:04.271243 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:04.271469 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:04.272055 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:04.272064 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:04.272987 (+ 923us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:04.273972 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.273997 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.275920 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.275929 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.278681 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.278703 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.280182 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.280192 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.280384 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.281484 (+ 1100us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:04.281520 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:04.282995 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:04.287498 (+ 4503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:04.287503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:04.287507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:04.288040 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:04.288514 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:04.288520 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:04.288524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:04.288669 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:04.289299 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:04.289304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:04.289545 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 22:02:04.289697 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 22:02:04.290066 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":644,"threads_started":1}
I20260927 22:02:04.290908 11343 ts_manager.cc:194] Registered new tserver with Master: 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265)
I20260927 22:02:04.292102 11343 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:55761
I20260927 22:02:04.292562 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 11695
I20260927 22:02:04.292783 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:02:04.293321 3708 external_mini_cluster.cc:1428] Setting key 79695b5a7e78297c9837b391b8d55abb
I20260927 22:02:04.297298 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 22:02:04 dist-test-slave-sg4j krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 22:02:04 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:04 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:02:04.361819 11719 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:04.348287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.3.159.62:39931)
0927 22:02:04.348584 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:04.348592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:04.348925 (+ 333us) server_negotiation.cc:407] Connection header received
0927 22:02:04.349139 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:04.349145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:04.349289 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:04.349519 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:04.351391 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.352351 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.353841 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.354400 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.354541 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.361441 (+ 6900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40322: BlockingRecv error: recv got EOF from 127.0.0.1:40322 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 27 22:02:04 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:02:04.384727 11719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.366690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40330 (local address 127.3.159.62:39931)
0927 22:02:04.367035 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:04.367044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:04.367420 (+ 376us) server_negotiation.cc:407] Connection header received
0927 22:02:04.367697 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:04.367707 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:04.367828 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:04.367968 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:04.369697 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.370476 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.371793 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.372244 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.372416 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.377234 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:04.377247 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:04.377252 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:04.377299 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:04.381187 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:04.382247 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:04.382252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:04.382308 (+ 56us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:04.382489 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:04.383002 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:04.383007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:04.383009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:04.383408 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:04.383642 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:04.383993 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 22:02:04.384227 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 22:02:04.390005 11343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:02:04.397207 11343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:02:04.429796 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.414726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.3.159.1:34009)
0927 22:02:04.415557 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:04.415583 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:04.416489 (+ 906us) server_negotiation.cc:407] Connection header received
0927 22:02:04.417057 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:04.417072 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:04.417526 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:04.417758 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:04.419293 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.421024 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.426142 (+ 5118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.428350 (+ 2208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.428515 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.428762 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:04.429251 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 22:02:04.429497 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":430,"threads_started":1}
I20260927 22:02:04.430088 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.414486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:34009 (local address 127.0.0.1:53730)
0927 22:02:04.416285 (+ 1799us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:04.416356 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:04.416697 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:04.418209 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:04.418221 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:04.418288 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:04.418960 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.418980 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.421257 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.421269 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.425349 (+ 4080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.425422 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.428585 (+ 3163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.428593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.428877 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.428954 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 22:02:04.429219 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 22:02:04.429523 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1476,"mutex_wait_us":4,"thread_start_us":5455,"threads_started":1}
I20260927 22:02:04.434396 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.415477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.3.159.2:33197)
0927 22:02:04.419707 (+ 4230us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:04.419717 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:04.420422 (+ 705us) server_negotiation.cc:407] Connection header received
0927 22:02:04.420766 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:04.420776 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:04.421091 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:04.421320 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:04.422702 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.425122 (+ 2420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.430852 (+ 5730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.432449 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.432643 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.432952 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:04.433801 (+ 849us) server_negotiation.cc:299] Negotiation successful
0927 22:02:04.434038 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4055,"thread_start_us":3898,"threads_started":1}
I20260927 22:02:04.434427 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.414805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:33197 (local address 127.0.0.1:49352)
0927 22:02:04.420289 (+ 5484us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:04.420318 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:04.420541 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:04.421537 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:04.421542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:04.421574 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:04.422392 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.422415 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.427737 (+ 5322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.427743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.430467 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.430485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.433156 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.433165 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.433472 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.433510 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:02:04.433690 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 22:02:04.433896 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5308,"thread_start_us":978,"threads_started":1}
I20260927 22:02:04.435200 11481 tablet_service.cc:1467] Processing CreateTablet for tablet 6a33e21b21c849819eae227735a7789c (DEFAULT_TABLE table=test-table [id=f1e8ac3fc82045c18ba84ce1a1b6dc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:04.437002 11481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a33e21b21c849819eae227735a7789c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:04.439221 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.421776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:44265 (local address 127.0.0.1:40824)
0927 22:02:04.428156 (+ 6380us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:04.428189 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:04.428412 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:04.430095 (+ 1683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:04.430101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:04.430142 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:04.430597 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.430613 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.433022 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.433029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.435438 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.435462 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.437601 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.437610 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.437852 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.437978 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 22:02:04.438495 (+ 517us) client_negotiation.cc:240] Negotiation successful
0927 22:02:04.438750 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6169,"thread_start_us":6072,"threads_started":1}
I20260927 22:02:04.440416 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.420896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.3.159.3:44265)
0927 22:02:04.427817 (+ 6921us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:04.427830 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:04.428372 (+ 542us) server_negotiation.cc:407] Connection header received
0927 22:02:04.428686 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:04.428697 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:04.428984 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:04.429221 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:04.430887 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.432708 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.435706 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.437266 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.437470 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.438056 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:04.439748 (+ 1692us) server_negotiation.cc:299] Negotiation successful
0927 22:02:04.440013 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6645,"thread_start_us":6373,"threads_started":1}
I20260927 22:02:04.439909 11626 tablet_service.cc:1467] Processing CreateTablet for tablet 6a33e21b21c849819eae227735a7789c (DEFAULT_TABLE table=test-table [id=f1e8ac3fc82045c18ba84ce1a1b6dc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:04.442025 11626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a33e21b21c849819eae227735a7789c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:04.445902 11761 tablet_service.cc:1467] Processing CreateTablet for tablet 6a33e21b21c849819eae227735a7789c (DEFAULT_TABLE table=test-table [id=f1e8ac3fc82045c18ba84ce1a1b6dc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:04.448055 11761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a33e21b21c849819eae227735a7789c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:04.451983 11855 tablet_bootstrap.cc:492] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: Bootstrap starting.
I20260927 22:02:04.456024 11855 tablet_bootstrap.cc:654] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:04.458166 11855 log.cc:824] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: Log is configured to *not* fsync() on all Append() calls
I20260927 22:02:04.461078 11857 tablet_bootstrap.cc:492] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: Bootstrap starting.
I20260927 22:02:04.461959 11855 tablet_bootstrap.cc:492] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: No bootstrap required, opened a new log
I20260927 22:02:04.462281 11855 ts_tablet_manager.cc:1397] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 22:02:04.466220 11857 tablet_bootstrap.cc:654] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:04.466676 11858 tablet_bootstrap.cc:492] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: Bootstrap starting.
I20260927 22:02:04.468225 11857 log.cc:824] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: Log is configured to *not* fsync() on all Append() calls
I20260927 22:02:04.471542 11858 tablet_bootstrap.cc:654] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:04.472820 11857 tablet_bootstrap.cc:492] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: No bootstrap required, opened a new log
I20260927 22:02:04.473196 11857 ts_tablet_manager.cc:1397] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260927 22:02:04.473243 11858 log.cc:824] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:02:04.472998 11855 raft_consensus.cc:348] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:04.473497 11855 raft_consensus.cc:374] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:04.473594 11855 raft_consensus.cc:729] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9a8f4a2b7f47afb5790bba81fec074, State: Initialized, Role: FOLLOWER
I20260927 22:02:04.474329 11855 consensus_queue.cc:260] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:04.476452 11855 ts_tablet_manager.cc:1428] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260927 22:02:04.477583 11858 tablet_bootstrap.cc:492] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: No bootstrap required, opened a new log
I20260927 22:02:04.477946 11858 ts_tablet_manager.cc:1397] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 22:02:04.486155 11857 raft_consensus.cc:348] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:04.486872 11857 raft_consensus.cc:374] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:04.487005 11857 raft_consensus.cc:729] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6b55eeb6d74baa97e4f23497e03b01, State: Initialized, Role: FOLLOWER
I20260927 22:02:04.487960 11857 consensus_queue.cc:260] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:04.489508 11691 heartbeater.cc:503] Master 127.3.159.62:39931 was elected leader, sending a full tablet report...
I20260927 22:02:04.490536 11857 ts_tablet_manager.cc:1428] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: Time spent starting tablet: real 0.017s user 0.015s sys 0.004s
I20260927 22:02:04.490602 11858 raft_consensus.cc:348] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:04.491175 11858 raft_consensus.cc:374] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:04.491312 11858 raft_consensus.cc:729] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a16926ad454239b4914139056e2a2a, State: Initialized, Role: FOLLOWER
I20260927 22:02:04.492125 11858 consensus_queue.cc:260] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:04.493628 11831 heartbeater.cc:503] Master 127.3.159.62:39931 was elected leader, sending a full tablet report...
I20260927 22:02:04.494719 11858 ts_tablet_manager.cc:1428] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260927 22:02:04.536578 11832 tablet.cc:2369] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:02:04.612421 11692 tablet.cc:2369] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:02:04.720420 11552 tablet.cc:2369] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:02:04.772754 11861 raft_consensus.cc:482] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:02:04.773113 11861 raft_consensus.cc:504] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:04.775034 11861 leader_election.cc:288] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2:33197), 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265)
I20260927 22:02:04.783536 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.775541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:48421 (local address 127.3.159.2:33197)
0927 22:02:04.775830 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:04.775839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:04.776677 (+ 838us) server_negotiation.cc:407] Connection header received
0927 22:02:04.776945 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:04.776950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:04.777020 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:04.777126 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:04.778014 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.778915 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.781169 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.782113 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.782292 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.782427 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:04.782866 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 22:02:04.783017 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 22:02:04.783638 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.775710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:56269 (local address 127.3.159.3:44265)
0927 22:02:04.776011 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:04.776022 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:04.776738 (+ 716us) server_negotiation.cc:407] Connection header received
0927 22:02:04.776959 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:04.776965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:04.777037 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:04.777148 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:04.778068 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.778874 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.781165 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.782042 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.782446 (+ 404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.782574 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:04.782909 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 22:02:04.783170 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 22:02:04.783821 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.775511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:44265 (local address 127.3.159.1:56269)
0927 22:02:04.776587 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:04.776605 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:04.776810 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:04.777179 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:04.777183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:04.777210 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:04.777897 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.777911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.779013 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.779018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.780964 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.780978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.782529 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.782535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.782729 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.782758 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:02:04.782881 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:02:04.783194 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":22,"thread_start_us":726,"threads_started":1}
I20260927 22:02:04.784443 11864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.775447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:33197 (local address 127.3.159.1:48421)
0927 22:02:04.776515 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:04.776541 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:04.776775 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:04.777141 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:04.777146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:04.777170 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:04.777857 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.777870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.779068 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.779076 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.780970 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:04.780985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.782390 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.782397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:04.782668 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.782698 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 22:02:04.782826 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 22:02:04.783151 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":314,"thread_start_us":526,"threads_started":1}
I20260927 22:02:04.785317 11784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a33e21b21c849819eae227735a7789c" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a16926ad454239b4914139056e2a2a" is_pre_election: true
I20260927 22:02:04.786098 11784 raft_consensus.cc:2454] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 0.
I20260927 22:02:04.786001 11646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a33e21b21c849819eae227735a7789c" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" is_pre_election: true
I20260927 22:02:04.786688 11646 raft_consensus.cc:2454] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 0.
I20260927 22:02:04.787146 11433 leader_election.cc:302] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a16926ad454239b4914139056e2a2a, cc9a8f4a2b7f47afb5790bba81fec074; no voters:
I20260927 22:02:04.787720 11861 raft_consensus.cc:2781] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:02:04.787973 11861 raft_consensus.cc:482] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:02:04.788097 11861 raft_consensus.cc:3037] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:04.790599 11861 raft_consensus.cc:504] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:04.791381 11861 leader_election.cc:288] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 election: Requested vote from peers 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2:33197), 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265)
I20260927 22:02:04.792043 11646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a33e21b21c849819eae227735a7789c" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6b55eeb6d74baa97e4f23497e03b01"
I20260927 22:02:04.792161 11784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a33e21b21c849819eae227735a7789c" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a16926ad454239b4914139056e2a2a"
I20260927 22:02:04.792287 11646 raft_consensus.cc:3037] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:04.792388 11784 raft_consensus.cc:3037] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:04.795629 11646 raft_consensus.cc:2454] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 1.
I20260927 22:02:04.795641 11784 raft_consensus.cc:2454] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 1.
I20260927 22:02:04.796281 11433 leader_election.cc:302] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 63a16926ad454239b4914139056e2a2a, 9f6b55eeb6d74baa97e4f23497e03b01, cc9a8f4a2b7f47afb5790bba81fec074; no voters:
I20260927 22:02:04.796746 11861 raft_consensus.cc:2781] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:02:04.797415 11861 raft_consensus.cc:686] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 LEADER]: Becoming Leader. State: Replica: cc9a8f4a2b7f47afb5790bba81fec074, State: Running, Role: LEADER
I20260927 22:02:04.798223 11861 consensus_queue.cc:237] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:04.804167 11344 catalog_manager.cc:5495] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1). New cstate: current_term: 1 leader_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } health_report { overall_health: UNKNOWN } } }
I20260927 22:02:04.891996 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:04.885505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.3.159.1:34009)
0927 22:02:04.885785 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:04.885792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:04.886039 (+ 247us) server_negotiation.cc:407] Connection header received
0927 22:02:04.886361 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:04.886366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:04.886471 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:04.886652 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:02:04.887110 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.887892 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.889332 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:04.889929 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:04.890074 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:04.890553 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:02:04.891295 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:02:04.891466 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:04.891580 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 22:02:04.891678 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 22:02:04.902158 11646 raft_consensus.cc:1261] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Refusing update from remote peer cc9a8f4a2b7f47afb5790bba81fec074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:04.902158 11784 raft_consensus.cc:1261] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Refusing update from remote peer cc9a8f4a2b7f47afb5790bba81fec074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:04.903581 11866 consensus_queue.cc:1035] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:04.904213 11861 consensus_queue.cc:1035] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:04.926043 11872 mvcc.cc:205] Tried to move back new op lower bound from 7334078565986557952 to 7334078565578694656. Current Snapshot: MvccSnapshot[applied={T|T < 7334078565986557952}]
I20260927 22:02:04.944577 11344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:02:04.944983 11344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:02:04.950443 11344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853: Sending DeleteTablet for 3 replicas of tablet 6a33e21b21c849819eae227735a7789c
I20260927 22:02:04.952608 11626 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a33e21b21c849819eae227735a7789c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:04 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:49352
I20260927 22:02:04.952605 11761 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a33e21b21c849819eae227735a7789c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:04 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:40824
I20260927 22:02:04.953986 11481 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a33e21b21c849819eae227735a7789c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:04 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:53730
I20260927 22:02:04.955770 11344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:02:04.957058 11344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:02:04.959610 11881 tablet_replica.cc:330] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: stopping tablet replica
I20260927 22:02:04.960809 11881 raft_consensus.cc:2229] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:02:04.961750 11881 raft_consensus.cc:2258] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:04.963261 11880 tablet_replica.cc:330] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: stopping tablet replica
I20260927 22:02:04.963605 11882 tablet_replica.cc:330] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: stopping tablet replica
I20260927 22:02:04.964169 11882 raft_consensus.cc:2229] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:02:04.964181 11880 raft_consensus.cc:2229] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:02:04.965010 11880 raft_consensus.cc:2258] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:04.965000 11882 raft_consensus.cc:2258] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:04.967156 11882 ts_tablet_manager.cc:1905] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:04.971089 11881 ts_tablet_manager.cc:1905] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:04.971647 11880 ts_tablet_manager.cc:1905] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:04.972448 11626 tablet_service.cc:1467] Processing CreateTablet for tablet 22bb990fa550475aae50b53c0b1ee277 (DEFAULT_TABLE table=test-table [id=4b026a952297499d80260d42088897e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:04.972996 11626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bb990fa550475aae50b53c0b1ee277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:04.980150 11481 tablet_service.cc:1467] Processing CreateTablet for tablet 22bb990fa550475aae50b53c0b1ee277 (DEFAULT_TABLE table=test-table [id=4b026a952297499d80260d42088897e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:04.980767 11481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bb990fa550475aae50b53c0b1ee277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:04.982189 11880 ts_tablet_manager.cc:1918] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:02:04.982527 11880 log.cc:1196] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/6a33e21b21c849819eae227735a7789c
I20260927 22:02:04.983366 11880 ts_tablet_manager.cc:1939] T 6a33e21b21c849819eae227735a7789c P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting consensus metadata
I20260927 22:02:04.983912 11882 ts_tablet_manager.cc:1918] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:02:04.984149 11882 log.cc:1196] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/6a33e21b21c849819eae227735a7789c
I20260927 22:02:04.984689 11882 ts_tablet_manager.cc:1939] T 6a33e21b21c849819eae227735a7789c P cc9a8f4a2b7f47afb5790bba81fec074: Deleting consensus metadata
I20260927 22:02:04.986482 11328 catalog_manager.cc:4841] TS 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2:33197): tablet 6a33e21b21c849819eae227735a7789c (table test-table [id=f1e8ac3fc82045c18ba84ce1a1b6dc55]) successfully deleted
I20260927 22:02:04.986652 11330 catalog_manager.cc:4841] TS cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1:34009): tablet 6a33e21b21c849819eae227735a7789c (table test-table [id=f1e8ac3fc82045c18ba84ce1a1b6dc55]) successfully deleted
I20260927 22:02:04.987720 11761 tablet_service.cc:1467] Processing CreateTablet for tablet 22bb990fa550475aae50b53c0b1ee277 (DEFAULT_TABLE table=test-table [id=4b026a952297499d80260d42088897e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:04.988474 11761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22bb990fa550475aae50b53c0b1ee277. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:04.988829 11881 ts_tablet_manager.cc:1918] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:02:04.989037 11881 log.cc:1196] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/6a33e21b21c849819eae227735a7789c
I20260927 22:02:04.989614 11881 ts_tablet_manager.cc:1939] T 6a33e21b21c849819eae227735a7789c P 63a16926ad454239b4914139056e2a2a: Deleting consensus metadata
I20260927 22:02:04.991600 11330 catalog_manager.cc:4841] TS 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265): tablet 6a33e21b21c849819eae227735a7789c (table test-table [id=f1e8ac3fc82045c18ba84ce1a1b6dc55]) successfully deleted
I20260927 22:02:04.998281 11883 tablet_bootstrap.cc:492] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01: Bootstrap starting.
I20260927 22:02:05.001469 11883 tablet_bootstrap.cc:654] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.006035 11884 tablet_bootstrap.cc:492] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a: Bootstrap starting.
I20260927 22:02:05.008248 11884 tablet_bootstrap.cc:654] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.011853 11885 tablet_bootstrap.cc:492] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074: Bootstrap starting.
I20260927 22:02:05.012904 11883 tablet_bootstrap.cc:492] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01: No bootstrap required, opened a new log
I20260927 22:02:05.013091 11883 ts_tablet_manager.cc:1397] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20260927 22:02:05.013957 11883 raft_consensus.cc:348] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.014194 11883 raft_consensus.cc:374] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.014279 11883 raft_consensus.cc:729] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6b55eeb6d74baa97e4f23497e03b01, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.014523 11883 consensus_queue.cc:260] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.014973 11885 tablet_bootstrap.cc:654] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.015213 11883 ts_tablet_manager.cc:1428] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 22:02:05.020493 11884 tablet_bootstrap.cc:492] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a: No bootstrap required, opened a new log
I20260927 22:02:05.020766 11884 ts_tablet_manager.cc:1397] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260927 22:02:05.022119 11884 raft_consensus.cc:348] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.022514 11884 raft_consensus.cc:374] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.022614 11884 raft_consensus.cc:729] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a16926ad454239b4914139056e2a2a, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.023047 11884 consensus_queue.cc:260] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.023906 11884 ts_tablet_manager.cc:1428] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 22:02:05.024528 11885 tablet_bootstrap.cc:492] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074: No bootstrap required, opened a new log
I20260927 22:02:05.024719 11885 ts_tablet_manager.cc:1397] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260927 22:02:05.025673 11885 raft_consensus.cc:348] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.025954 11885 raft_consensus.cc:374] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.026065 11885 raft_consensus.cc:729] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9a8f4a2b7f47afb5790bba81fec074, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.026348 11885 consensus_queue.cc:260] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.026978 11885 ts_tablet_manager.cc:1428] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:02:05.051826 11866 raft_consensus.cc:482] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:02:05.052433 11866 raft_consensus.cc:504] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.053324 11866 leader_election.cc:288] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2:33197), 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265)
I20260927 22:02:05.054095 11646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bb990fa550475aae50b53c0b1ee277" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" is_pre_election: true
I20260927 22:02:05.054472 11646 raft_consensus.cc:2454] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 0.
I20260927 22:02:05.054550 11784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bb990fa550475aae50b53c0b1ee277" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a16926ad454239b4914139056e2a2a" is_pre_election: true
I20260927 22:02:05.054872 11784 raft_consensus.cc:2454] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 0.
I20260927 22:02:05.055079 11431 leader_election.cc:302] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f6b55eeb6d74baa97e4f23497e03b01, cc9a8f4a2b7f47afb5790bba81fec074; no voters:
I20260927 22:02:05.056036 11866 raft_consensus.cc:2781] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:02:05.056221 11866 raft_consensus.cc:482] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:02:05.056346 11866 raft_consensus.cc:3037] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.059731 11866 raft_consensus.cc:504] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.060554 11866 leader_election.cc:288] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 election: Requested vote from peers 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2:33197), 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265)
I20260927 22:02:05.061440 11784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bb990fa550475aae50b53c0b1ee277" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a16926ad454239b4914139056e2a2a"
I20260927 22:02:05.061539 11646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22bb990fa550475aae50b53c0b1ee277" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6b55eeb6d74baa97e4f23497e03b01"
I20260927 22:02:05.061718 11784 raft_consensus.cc:3037] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.061882 11646 raft_consensus.cc:3037] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.065455 11784 raft_consensus.cc:2454] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 1.
I20260927 22:02:05.065500 11646 raft_consensus.cc:2454] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 1.
I20260927 22:02:05.066316 11433 leader_election.cc:302] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a16926ad454239b4914139056e2a2a, cc9a8f4a2b7f47afb5790bba81fec074; no voters:
I20260927 22:02:05.066776 11866 raft_consensus.cc:2781] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:02:05.067097 11866 raft_consensus.cc:686] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 LEADER]: Becoming Leader. State: Replica: cc9a8f4a2b7f47afb5790bba81fec074, State: Running, Role: LEADER
I20260927 22:02:05.067408 11866 consensus_queue.cc:237] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.071310 11343 catalog_manager.cc:5495] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1). New cstate: current_term: 1 leader_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } health_report { overall_health: UNKNOWN } } }
Sep 27 22:02:05 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546522, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.62@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:02:05.142625 11896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.124689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.3.159.62:39931)
0927 22:02:05.125965 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:05.125970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:05.125996 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:02:05.126054 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:05.126058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:05.126130 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:05.126264 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:05.127944 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.128848 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.130925 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.131427 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.131546 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.136214 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:05.136220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:05.136223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:05.136314 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:05.139485 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:05.140314 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:05.140319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:05.140328 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:05.140444 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:05.140811 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:05.140816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:05.140819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:05.141132 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:05.141290 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:05.141811 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 22:02:05.142040 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166,"thread_start_us":990,"threads_started":1}
I20260927 22:02:05.142652 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.124574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:39931 (local address 127.0.0.1:40334)
0927 22:02:05.125568 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:05.125595 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:05.125791 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:05.126358 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:05.126365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:05.127363 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:05.127704 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.127718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.129035 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.129041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.130726 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.130743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.131642 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.131650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.131855 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.132739 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:05.132763 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:05.135952 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:05.139659 (+ 3707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:05.139682 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:05.139692 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:05.140177 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:05.140557 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:05.140561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:05.140565 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:05.140702 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:05.141269 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:05.141278 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:05.141529 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 22:02:05.141722 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 22:02:05.142177 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":144,"thread_start_us":494,"threads_started":1}
I20260927 22:02:05.149075 11343 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:40334:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 22:02:05.150444 11343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:02:05.160568 11481 tablet_service.cc:1467] Processing CreateTablet for tablet 8f70278dc4934d4fa1110b483f8b6265 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9e3c25ac36043649c42226bbcbb9e04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:02:05.161320 11481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f70278dc4934d4fa1110b483f8b6265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:05.166759 11626 tablet_service.cc:1467] Processing CreateTablet for tablet 8f70278dc4934d4fa1110b483f8b6265 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9e3c25ac36043649c42226bbcbb9e04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:02:05.166872 11761 tablet_service.cc:1467] Processing CreateTablet for tablet 8f70278dc4934d4fa1110b483f8b6265 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9e3c25ac36043649c42226bbcbb9e04]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 22:02:05.167455 11626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f70278dc4934d4fa1110b483f8b6265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:05.167729 11761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f70278dc4934d4fa1110b483f8b6265. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:05.176165 11884 tablet_bootstrap.cc:492] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a: Bootstrap starting.
I20260927 22:02:05.176199 11883 tablet_bootstrap.cc:492] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: Bootstrap starting.
I20260927 22:02:05.178675 11883 tablet_bootstrap.cc:654] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.178715 11884 tablet_bootstrap.cc:654] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.182245 11885 tablet_bootstrap.cc:492] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074: Bootstrap starting.
I20260927 22:02:05.184404 11885 tablet_bootstrap.cc:654] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.194609 11885 tablet_bootstrap.cc:492] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074: No bootstrap required, opened a new log
I20260927 22:02:05.195017 11885 ts_tablet_manager.cc:1397] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.006s
I20260927 22:02:05.195928 11885 raft_consensus.cc:348] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.196264 11885 raft_consensus.cc:374] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.196355 11885 raft_consensus.cc:729] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9a8f4a2b7f47afb5790bba81fec074, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.196686 11885 consensus_queue.cc:260] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.197206 11866 tablet_replica.cc:439] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } } }
I20260927 22:02:05.197517 11866 tablet_replica.cc:442] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:02:05.197628 11884 tablet_bootstrap.cc:492] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a: No bootstrap required, opened a new log
I20260927 22:02:05.197911 11884 ts_tablet_manager.cc:1397] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.012s
I20260927 22:02:05.198700 11885 ts_tablet_manager.cc:1428] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260927 22:02:05.199095 11884 raft_consensus.cc:348] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.199532 11884 raft_consensus.cc:374] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.199676 11884 raft_consensus.cc:729] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a16926ad454239b4914139056e2a2a, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.200306 11884 consensus_queue.cc:260] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.201057 11884 ts_tablet_manager.cc:1428] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 22:02:05.201870 11863 tablet_replica.cc:439] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } } }
I20260927 22:02:05.202333 11863 tablet_replica.cc:442] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:02:05.205853 11883 tablet_bootstrap.cc:492] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: No bootstrap required, opened a new log
I20260927 22:02:05.206075 11883 ts_tablet_manager.cc:1397] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.007s
I20260927 22:02:05.206923 11883 raft_consensus.cc:348] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.207202 11883 raft_consensus.cc:374] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.207294 11883 raft_consensus.cc:729] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6b55eeb6d74baa97e4f23497e03b01, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.207554 11883 consensus_queue.cc:260] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.208070 11862 tablet_replica.cc:439] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } } }
I20260927 22:02:05.208493 11862 tablet_replica.cc:442] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:02:05.209444 11883 ts_tablet_manager.cc:1428] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 22:02:05.254741 11862 raft_consensus.cc:482] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:02:05.255030 11862 raft_consensus.cc:504] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.257005 11862 leader_election.cc:288] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1:34009), 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265)
I20260927 22:02:05.267318 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.257927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:59909 (local address 127.3.159.1:34009)
0927 22:02:05.258129 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:05.258134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:05.258707 (+ 573us) server_negotiation.cc:407] Connection header received
0927 22:02:05.259152 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:05.259159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:05.259254 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:05.259324 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:05.260328 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.261618 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.264655 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.266054 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.266234 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.266348 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:05.266764 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 22:02:05.266915 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 22:02:05.267421 11900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.257499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:34009 (local address 127.3.159.2:59909)
0927 22:02:05.258519 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:05.258547 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:05.258954 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:05.259564 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:05.259569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:05.259601 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:05.260094 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.260109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.261787 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.261793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.264365 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.264387 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.266326 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.266334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.266534 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.266590 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:02:05.266713 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:02:05.266950 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":91,"thread_start_us":616,"threads_started":1}
I20260927 22:02:05.267884 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.258954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:34999 (local address 127.3.159.3:44265)
0927 22:02:05.259228 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:05.259237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:05.259955 (+ 718us) server_negotiation.cc:407] Connection header received
0927 22:02:05.260206 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:05.260214 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:05.260321 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:05.260426 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:05.261083 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.262107 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.265043 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.266206 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.266347 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.266459 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:05.266998 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 22:02:05.267173 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:02:05.268625 11901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.258679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:44265 (local address 127.3.159.2:34999)
0927 22:02:05.259811 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:05.259833 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:05.260047 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:05.260484 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:05.260490 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:05.260530 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:05.260879 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.260891 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.262272 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.262278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.264825 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.264843 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.266610 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.266615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.266826 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.266855 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:02:05.268177 (+ 1322us) client_negotiation.cc:240] Negotiation successful
0927 22:02:05.268312 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":61,"thread_start_us":745,"threads_started":1}
I20260927 22:02:05.268455 11501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f70278dc4934d4fa1110b483f8b6265" candidate_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" is_pre_election: true
I20260927 22:02:05.268929 11501 raft_consensus.cc:2454] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f6b55eeb6d74baa97e4f23497e03b01 in term 0.
I20260927 22:02:05.269567 11784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f70278dc4934d4fa1110b483f8b6265" candidate_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a16926ad454239b4914139056e2a2a" is_pre_election: true
I20260927 22:02:05.269711 11573 leader_election.cc:302] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f6b55eeb6d74baa97e4f23497e03b01, cc9a8f4a2b7f47afb5790bba81fec074; no voters:
I20260927 22:02:05.269861 11784 raft_consensus.cc:2454] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f6b55eeb6d74baa97e4f23497e03b01 in term 0.
I20260927 22:02:05.270335 11862 raft_consensus.cc:2781] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:02:05.270494 11862 raft_consensus.cc:482] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:02:05.270648 11862 raft_consensus.cc:3037] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.273911 11862 raft_consensus.cc:504] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.274703 11862 leader_election.cc:288] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [CANDIDATE]: Term 1 election: Requested vote from peers cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1:34009), 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265)
I20260927 22:02:05.275590 11501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f70278dc4934d4fa1110b483f8b6265" candidate_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9a8f4a2b7f47afb5790bba81fec074"
I20260927 22:02:05.275605 11784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f70278dc4934d4fa1110b483f8b6265" candidate_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a16926ad454239b4914139056e2a2a"
I20260927 22:02:05.275930 11784 raft_consensus.cc:3037] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.275916 11501 raft_consensus.cc:3037] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.279516 11784 raft_consensus.cc:2454] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6b55eeb6d74baa97e4f23497e03b01 in term 1.
I20260927 22:02:05.279683 11501 raft_consensus.cc:2454] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6b55eeb6d74baa97e4f23497e03b01 in term 1.
I20260927 22:02:05.280208 11573 leader_election.cc:302] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a16926ad454239b4914139056e2a2a, 9f6b55eeb6d74baa97e4f23497e03b01; no voters:
I20260927 22:02:05.280646 11862 raft_consensus.cc:2781] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:02:05.281520 11862 raft_consensus.cc:686] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 LEADER]: Becoming Leader. State: Replica: 9f6b55eeb6d74baa97e4f23497e03b01, State: Running, Role: LEADER
I20260927 22:02:05.282373 11862 consensus_queue.cc:237] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.286383 11902 tablet_replica.cc:439] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: TxnStatusTablet state changed. Reason: New leader 9f6b55eeb6d74baa97e4f23497e03b01. Latest consensus state: current_term: 1 leader_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } } }
I20260927 22:02:05.286846 11902 tablet_replica.cc:442] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:02:05.287192 11344 catalog_manager.cc:5495] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2). New cstate: current_term: 1 leader_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } health_report { overall_health: UNKNOWN } } }
I20260927 22:02:05.293968 11904 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 22:02:05.294286 11904 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 22:02:05.334260 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.322303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:33197 (local address 127.0.0.1:49356)
0927 22:02:05.322653 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:05.322682 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:05.322893 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:05.326702 (+ 3809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:05.326709 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:05.326731 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:02:05.326971 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.326985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.328786 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.328792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.330466 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.330485 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.331679 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.331688 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.331906 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.332004 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:02:05.333507 (+ 1503us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:02:05.333534 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 22:02:05.333658 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 22:02:05.333852 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 22:02:05.334368 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.322799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.3.159.2:33197)
0927 22:02:05.325999 (+ 3200us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:05.326007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:05.326040 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:02:05.326185 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:05.326191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:05.326360 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:05.326580 (+ 220us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:02:05.327220 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.328590 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.330668 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.331449 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.331606 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.332327 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:02:05.333347 (+ 1020us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:02:05.333556 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:05.333713 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 22:02:05.333823 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3015,"thread_start_us":2828,"threads_started":1}
I20260927 22:02:05.340338 11784 raft_consensus.cc:1261] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Refusing update from remote peer 9f6b55eeb6d74baa97e4f23497e03b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:05.340778 11501 raft_consensus.cc:1261] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Refusing update from remote peer 9f6b55eeb6d74baa97e4f23497e03b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:05.341367 11862 consensus_queue.cc:1035] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:05.342191 11862 consensus_queue.cc:1035] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:05.354497 11863 tablet_replica.cc:439] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a: TxnStatusTablet state changed. Reason: New leader 9f6b55eeb6d74baa97e4f23497e03b01. Latest consensus state: current_term: 1 leader_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } } }
I20260927 22:02:05.355033 11863 tablet_replica.cc:442] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:02:05.361941 11866 tablet_replica.cc:439] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074: TxnStatusTablet state changed. Reason: New leader 9f6b55eeb6d74baa97e4f23497e03b01. Latest consensus state: current_term: 1 leader_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } } }
I20260927 22:02:05.364066 11902 tablet_replica.cc:439] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } } }
I20260927 22:02:05.364404 11902 tablet_replica.cc:442] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:02:05.368813 11863 tablet_replica.cc:439] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } } }
I20260927 22:02:05.369148 11863 tablet_replica.cc:442] T 8f70278dc4934d4fa1110b483f8b6265 P 63a16926ad454239b4914139056e2a2a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:02:05.371996 11866 tablet_replica.cc:442] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:02:05.377063 11862 tablet_replica.cc:439] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } } }
I20260927 22:02:05.377599 11862 tablet_replica.cc:442] T 8f70278dc4934d4fa1110b483f8b6265 P 9f6b55eeb6d74baa97e4f23497e03b01: This TxnStatusTablet replica's current role is: LEADER
I20260927 22:02:05.381865 11866 tablet_replica.cc:439] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } } }
I20260927 22:02:05.382241 11866 tablet_replica.cc:442] T 8f70278dc4934d4fa1110b483f8b6265 P cc9a8f4a2b7f47afb5790bba81fec074: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 22:02:05.414182 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.403711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:37357 (local address 127.3.159.2:33197)
0927 22:02:05.403902 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:05.403906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:05.406296 (+ 2390us) server_negotiation.cc:407] Connection header received
0927 22:02:05.406549 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:05.406556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:05.406691 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:05.406838 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:02:05.407643 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.408879 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.411537 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.412290 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.412495 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.413209 (+ 714us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:02:05.413370 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:02:05.413517 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:05.413718 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 22:02:05.413818 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:02:05.414362 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.403551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:33197 (local address 127.3.159.1:37357)
0927 22:02:05.406077 (+ 2526us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:05.406104 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:05.406357 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:05.406889 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:05.406895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:05.406910 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:02:05.407421 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.407434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.409077 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.409083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.411330 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.411348 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.412587 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.412595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.412826 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.412951 (+ 125us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:02:05.413533 (+ 582us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:02:05.413547 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:02:05.413647 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 22:02:05.413849 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2276,"thread_start_us":2076,"threads_started":1}
I20260927 22:02:05.428603 11784 raft_consensus.cc:1261] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Refusing update from remote peer cc9a8f4a2b7f47afb5790bba81fec074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:05.428715 11646 raft_consensus.cc:1261] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Refusing update from remote peer cc9a8f4a2b7f47afb5790bba81fec074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:05.429306 11878 consensus_queue.cc:1035] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:05.430284 11866 consensus_queue.cc:1035] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:05.448108 11870 mvcc.cc:205] Tried to move back new op lower bound from 7334078568147365888 to 7334078566679429120. Current Snapshot: MvccSnapshot[applied={T|T < 7334078568147365888}]
I20260927 22:02:05.451567 11871 mvcc.cc:205] Tried to move back new op lower bound from 7334078568147365888 to 7334078566679429120. Current Snapshot: MvccSnapshot[applied={T|T < 7334078568147365888}]
I20260927 22:02:05.505645 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.486795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:50461 (local address 127.3.159.1:34009)
0927 22:02:05.486988 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:05.486996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:05.497932 (+ 10936us) server_negotiation.cc:407] Connection header received
0927 22:02:05.498271 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:05.498276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:05.498390 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:05.498452 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:02:05.499216 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.500371 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.502994 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.503664 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.503766 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.504530 (+ 764us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:02:05.504741 (+ 211us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:02:05.504891 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:05.505095 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 22:02:05.505220 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:02:05.506006 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.486706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:34009 (local address 127.3.159.2:50461)
0927 22:02:05.497815 (+ 11109us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:05.497840 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:05.498070 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:05.498687 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:05.498692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:05.498705 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 22:02:05.499057 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.499066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.500532 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.500540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.502382 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:05.502401 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.503992 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.503998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:05.504194 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.504272 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 22:02:05.504907 (+ 635us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 22:02:05.504919 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 22:02:05.505038 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 22:02:05.505205 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10803,"thread_start_us":10687,"threads_started":1}
I20260927 22:02:05.579973 11344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:02:05.580312 11344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:02:05.584326 11344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853: Sending DeleteTablet for 3 replicas of tablet 22bb990fa550475aae50b53c0b1ee277
I20260927 22:02:05.585590 11761 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bb990fa550475aae50b53c0b1ee277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:05 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:40824
I20260927 22:02:05.585675 11481 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bb990fa550475aae50b53c0b1ee277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:05 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:53730
I20260927 22:02:05.586346 11626 tablet_service.cc:1514] Processing DeleteTablet for tablet 22bb990fa550475aae50b53c0b1ee277 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:05 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:49352
I20260927 22:02:05.601061 11921 tablet_replica.cc:330] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a: stopping tablet replica
I20260927 22:02:05.601416 11921 raft_consensus.cc:2229] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:02:05.601742 11921 raft_consensus.cc:2258] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:05.602028 11923 tablet_replica.cc:330] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01: stopping tablet replica
I20260927 22:02:05.602375 11923 raft_consensus.cc:2229] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:02:05.602767 11923 raft_consensus.cc:2258] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:05.603320 11921 ts_tablet_manager.cc:1905] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:05.604272 11923 ts_tablet_manager.cc:1905] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:05.605955 11922 tablet_replica.cc:330] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074: stopping tablet replica
I20260927 22:02:05.606248 11922 raft_consensus.cc:2229] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:02:05.606678 11922 raft_consensus.cc:2258] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:05.608178 11922 ts_tablet_manager.cc:1905] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:05.611971 11921 ts_tablet_manager.cc:1918] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:02:05.612119 11921 log.cc:1196] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/22bb990fa550475aae50b53c0b1ee277
I20260927 22:02:05.612618 11921 ts_tablet_manager.cc:1939] T 22bb990fa550475aae50b53c0b1ee277 P 63a16926ad454239b4914139056e2a2a: Deleting consensus metadata
I20260927 22:02:05.614102 11330 catalog_manager.cc:4841] TS 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265): tablet 22bb990fa550475aae50b53c0b1ee277 (table test-table [id=4b026a952297499d80260d42088897e5]) successfully deleted
I20260927 22:02:05.616652 11922 ts_tablet_manager.cc:1918] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:02:05.616814 11922 log.cc:1196] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/22bb990fa550475aae50b53c0b1ee277
I20260927 22:02:05.617295 11922 ts_tablet_manager.cc:1939] T 22bb990fa550475aae50b53c0b1ee277 P cc9a8f4a2b7f47afb5790bba81fec074: Deleting consensus metadata
I20260927 22:02:05.618721 11330 catalog_manager.cc:4841] TS cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1:34009): tablet 22bb990fa550475aae50b53c0b1ee277 (table test-table [id=4b026a952297499d80260d42088897e5]) successfully deleted
I20260927 22:02:05.622888 11923 ts_tablet_manager.cc:1918] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:02:05.623081 11923 log.cc:1196] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/22bb990fa550475aae50b53c0b1ee277
I20260927 22:02:05.623683 11923 ts_tablet_manager.cc:1939] T 22bb990fa550475aae50b53c0b1ee277 P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting consensus metadata
I20260927 22:02:05.625149 11328 catalog_manager.cc:4841] TS 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2:33197): tablet 22bb990fa550475aae50b53c0b1ee277 (table test-table [id=4b026a952297499d80260d42088897e5]) successfully deleted
Sep 27 22:02:05 dist-test-slave-sg4j krb5kdc[11299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 22:02:05 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:05 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:02:05.681627 11931 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:05.665960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.3.159.62:39931)
0927 22:02:05.667025 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:05.667031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:05.667090 (+ 59us) server_negotiation.cc:407] Connection header received
0927 22:02:05.667174 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:05.667178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:05.667269 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:05.667383 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:05.669606 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.670621 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.672714 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.673188 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.673330 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.681057 (+ 7727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40338: BlockingRecv error: recv got EOF from 127.0.0.1:40338 (error 108)
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":780,"threads_started":1}
Sep 27 22:02:05 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM
I20260927 22:02:05.709671 11931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.691855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.3.159.62:39931)
0927 22:02:05.692124 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:05.692133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:05.692739 (+ 606us) server_negotiation.cc:407] Connection header received
0927 22:02:05.692970 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:05.692975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:05.693069 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:05.693179 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:05.695021 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.695980 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.697984 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.698473 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.698627 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.703558 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:05.703564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:05.703567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:05.703617 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:05.706606 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:05.707444 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:05.707449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:05.707451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:05.707586 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:05.707999 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:05.708004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:05.708006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:05.708328 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:05.708647 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:05.708948 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 22:02:05.709215 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 22:02:05.714732 11344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:02:05.715580 11344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:02:05.725224 11481 tablet_service.cc:1467] Processing CreateTablet for tablet 943fd2fef44244d9ac0859155984d219 (DEFAULT_TABLE table=test-table [id=8337e00dc7034e7e9ed251d39c484f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:05.725531 11761 tablet_service.cc:1467] Processing CreateTablet for tablet 943fd2fef44244d9ac0859155984d219 (DEFAULT_TABLE table=test-table [id=8337e00dc7034e7e9ed251d39c484f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:05.725612 11626 tablet_service.cc:1467] Processing CreateTablet for tablet 943fd2fef44244d9ac0859155984d219 (DEFAULT_TABLE table=test-table [id=8337e00dc7034e7e9ed251d39c484f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:05.725808 11481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943fd2fef44244d9ac0859155984d219. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:05.726078 11761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943fd2fef44244d9ac0859155984d219. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:05.726094 11626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 943fd2fef44244d9ac0859155984d219. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:05.739007 11937 tablet_bootstrap.cc:492] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074: Bootstrap starting.
I20260927 22:02:05.739830 11939 tablet_bootstrap.cc:492] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01: Bootstrap starting.
I20260927 22:02:05.740082 11938 tablet_bootstrap.cc:492] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a: Bootstrap starting.
I20260927 22:02:05.741247 11937 tablet_bootstrap.cc:654] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.742987 11939 tablet_bootstrap.cc:654] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.746495 11938 tablet_bootstrap.cc:654] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.750859 11939 tablet_bootstrap.cc:492] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01: No bootstrap required, opened a new log
I20260927 22:02:05.751106 11939 ts_tablet_manager.cc:1397] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260927 22:02:05.751178 11937 tablet_bootstrap.cc:492] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074: No bootstrap required, opened a new log
I20260927 22:02:05.751415 11937 ts_tablet_manager.cc:1397] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260927 22:02:05.751925 11938 tablet_bootstrap.cc:492] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a: No bootstrap required, opened a new log
I20260927 22:02:05.752161 11938 ts_tablet_manager.cc:1397] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20260927 22:02:05.752259 11939 raft_consensus.cc:348] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.752583 11939 raft_consensus.cc:374] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.752705 11939 raft_consensus.cc:729] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6b55eeb6d74baa97e4f23497e03b01, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.753070 11939 consensus_queue.cc:260] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.753541 11938 raft_consensus.cc:348] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.753867 11938 raft_consensus.cc:374] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.754009 11938 raft_consensus.cc:729] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a16926ad454239b4914139056e2a2a, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.754177 11937 raft_consensus.cc:348] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.754532 11937 raft_consensus.cc:374] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.754387 11938 consensus_queue.cc:260] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.754678 11937 raft_consensus.cc:729] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9a8f4a2b7f47afb5790bba81fec074, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.754945 11937 consensus_queue.cc:260] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.755666 11937 ts_tablet_manager.cc:1428] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 22:02:05.756366 11938 ts_tablet_manager.cc:1428] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260927 22:02:05.757838 11939 ts_tablet_manager.cc:1428] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20260927 22:02:05.777599 11862 raft_consensus.cc:482] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:02:05.778047 11862 raft_consensus.cc:504] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.779181 11862 leader_election.cc:288] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1:34009), 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265)
I20260927 22:02:05.780351 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943fd2fef44244d9ac0859155984d219" candidate_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a16926ad454239b4914139056e2a2a" is_pre_election: true
I20260927 22:02:05.780402 11501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943fd2fef44244d9ac0859155984d219" candidate_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" is_pre_election: true
I20260927 22:02:05.780683 11783 raft_consensus.cc:2454] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f6b55eeb6d74baa97e4f23497e03b01 in term 0.
I20260927 22:02:05.780725 11501 raft_consensus.cc:2454] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f6b55eeb6d74baa97e4f23497e03b01 in term 0.
I20260927 22:02:05.781286 11573 leader_election.cc:302] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f6b55eeb6d74baa97e4f23497e03b01, cc9a8f4a2b7f47afb5790bba81fec074; no voters:
I20260927 22:02:05.781787 11862 raft_consensus.cc:2781] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:02:05.781963 11862 raft_consensus.cc:482] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:02:05.782130 11862 raft_consensus.cc:3037] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.785348 11862 raft_consensus.cc:504] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.786307 11862 leader_election.cc:288] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [CANDIDATE]: Term 1 election: Requested vote from peers cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1:34009), 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265)
I20260927 22:02:05.787117 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943fd2fef44244d9ac0859155984d219" candidate_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a16926ad454239b4914139056e2a2a"
I20260927 22:02:05.787164 11501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "943fd2fef44244d9ac0859155984d219" candidate_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9a8f4a2b7f47afb5790bba81fec074"
I20260927 22:02:05.787426 11783 raft_consensus.cc:3037] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.787499 11501 raft_consensus.cc:3037] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.790297 11783 raft_consensus.cc:2454] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6b55eeb6d74baa97e4f23497e03b01 in term 1.
I20260927 22:02:05.791023 11573 leader_election.cc:302] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a16926ad454239b4914139056e2a2a, 9f6b55eeb6d74baa97e4f23497e03b01; no voters:
I20260927 22:02:05.791039 11501 raft_consensus.cc:2454] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f6b55eeb6d74baa97e4f23497e03b01 in term 1.
I20260927 22:02:05.791428 11862 raft_consensus.cc:2781] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:02:05.791706 11862 raft_consensus.cc:686] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 LEADER]: Becoming Leader. State: Replica: 9f6b55eeb6d74baa97e4f23497e03b01, State: Running, Role: LEADER
I20260927 22:02:05.792093 11862 consensus_queue.cc:237] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.795801 11343 catalog_manager.cc:5495] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2). New cstate: current_term: 1 leader_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } health_report { overall_health: UNKNOWN } } }
I20260927 22:02:05.821130 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.814959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.3.159.2:33197)
0927 22:02:05.815214 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:05.815221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:05.815256 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:02:05.815510 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:05.815519 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:05.815627 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:05.815754 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:02:05.816116 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.817279 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.818870 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.819599 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.819750 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.820269 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:02:05.820424 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:02:05.820569 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:05.820696 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 22:02:05.820792 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 22:02:05.825477 11501 raft_consensus.cc:1261] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Refusing update from remote peer 9f6b55eeb6d74baa97e4f23497e03b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:05.825495 11783 raft_consensus.cc:1261] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Refusing update from remote peer 9f6b55eeb6d74baa97e4f23497e03b01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:05.826228 11862 consensus_queue.cc:1035] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:05.826821 11902 consensus_queue.cc:1035] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:05.848944 11343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:02:05.849216 11343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:02:05.853142 11343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853: Sending DeleteTablet for 3 replicas of tablet 943fd2fef44244d9ac0859155984d219
I20260927 22:02:05.854710 11481 tablet_service.cc:1514] Processing DeleteTablet for tablet 943fd2fef44244d9ac0859155984d219 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:05 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:53730
I20260927 22:02:05.854791 11761 tablet_service.cc:1514] Processing DeleteTablet for tablet 943fd2fef44244d9ac0859155984d219 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:05 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:40824
I20260927 22:02:05.855037 11922 tablet_replica.cc:330] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074: stopping tablet replica
I20260927 22:02:05.854363 11626 tablet_service.cc:1514] Processing DeleteTablet for tablet 943fd2fef44244d9ac0859155984d219 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:05 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:49352
I20260927 22:02:05.855245 11922 raft_consensus.cc:2229] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:02:05.855278 11921 tablet_replica.cc:330] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a: stopping tablet replica
I20260927 22:02:05.855324 11923 tablet_replica.cc:330] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01: stopping tablet replica
I20260927 22:02:05.855479 11922 raft_consensus.cc:2258] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:05.855521 11921 raft_consensus.cc:2229] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:02:05.855545 11923 raft_consensus.cc:2229] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:02:05.855792 11921 raft_consensus.cc:2258] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:05.856169 11923 raft_consensus.cc:2258] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:05.857429 11922 ts_tablet_manager.cc:1905] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:05.857975 11923 ts_tablet_manager.cc:1905] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:05.857147 11343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 22:02:05.862509 11921 ts_tablet_manager.cc:1905] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 22:02:05.863098 11343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:02:05.865088 11922 ts_tablet_manager.cc:1918] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:02:05.865290 11922 log.cc:1196] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/943fd2fef44244d9ac0859155984d219
I20260927 22:02:05.865839 11922 ts_tablet_manager.cc:1939] T 943fd2fef44244d9ac0859155984d219 P cc9a8f4a2b7f47afb5790bba81fec074: Deleting consensus metadata
I20260927 22:02:05.871259 11923 ts_tablet_manager.cc:1918] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:02:05.871546 11923 log.cc:1196] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/943fd2fef44244d9ac0859155984d219
I20260927 22:02:05.872149 11923 ts_tablet_manager.cc:1939] T 943fd2fef44244d9ac0859155984d219 P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting consensus metadata
I20260927 22:02:05.873852 11328 catalog_manager.cc:4841] TS 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2:33197): tablet 943fd2fef44244d9ac0859155984d219 (table test-table [id=8337e00dc7034e7e9ed251d39c484f2a]) successfully deleted
I20260927 22:02:05.874238 11481 tablet_service.cc:1467] Processing CreateTablet for tablet 01e663c0734c4f72bb886c7539a50236 (DEFAULT_TABLE table=test-table [id=2d13e7089ebf40d3be1125d1e7f2b299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:05.874166 11626 tablet_service.cc:1467] Processing CreateTablet for tablet 01e663c0734c4f72bb886c7539a50236 (DEFAULT_TABLE table=test-table [id=2d13e7089ebf40d3be1125d1e7f2b299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:05.874785 11626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e663c0734c4f72bb886c7539a50236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:05.874785 11481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e663c0734c4f72bb886c7539a50236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:05.875648 11761 tablet_service.cc:1467] Processing CreateTablet for tablet 01e663c0734c4f72bb886c7539a50236 (DEFAULT_TABLE table=test-table [id=2d13e7089ebf40d3be1125d1e7f2b299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:05.876233 11761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e663c0734c4f72bb886c7539a50236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:05.879187 11330 catalog_manager.cc:4841] TS cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1:34009): tablet 943fd2fef44244d9ac0859155984d219 (table test-table [id=8337e00dc7034e7e9ed251d39c484f2a]) successfully deleted
I20260927 22:02:05.882678 11939 tablet_bootstrap.cc:492] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01: Bootstrap starting.
I20260927 22:02:05.884869 11939 tablet_bootstrap.cc:654] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.885357 11921 ts_tablet_manager.cc:1918] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:02:05.885637 11921 log.cc:1196] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/943fd2fef44244d9ac0859155984d219
I20260927 22:02:05.886083 11921 ts_tablet_manager.cc:1939] T 943fd2fef44244d9ac0859155984d219 P 63a16926ad454239b4914139056e2a2a: Deleting consensus metadata
I20260927 22:02:05.887426 11330 catalog_manager.cc:4841] TS 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265): tablet 943fd2fef44244d9ac0859155984d219 (table test-table [id=8337e00dc7034e7e9ed251d39c484f2a]) successfully deleted
I20260927 22:02:05.888867 11938 tablet_bootstrap.cc:492] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a: Bootstrap starting.
I20260927 22:02:05.889148 11937 tablet_bootstrap.cc:492] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074: Bootstrap starting.
I20260927 22:02:05.890512 11938 tablet_bootstrap.cc:654] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.891732 11937 tablet_bootstrap.cc:654] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:05.896184 11937 tablet_bootstrap.cc:492] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074: No bootstrap required, opened a new log
I20260927 22:02:05.896400 11937 ts_tablet_manager.cc:1397] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 22:02:05.897580 11937 raft_consensus.cc:348] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.898003 11937 raft_consensus.cc:374] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.898104 11937 raft_consensus.cc:729] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9a8f4a2b7f47afb5790bba81fec074, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.900079 11937 consensus_queue.cc:260] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.902731 11937 ts_tablet_manager.cc:1428] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260927 22:02:05.904662 11938 tablet_bootstrap.cc:492] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a: No bootstrap required, opened a new log
I20260927 22:02:05.904672 11939 tablet_bootstrap.cc:492] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01: No bootstrap required, opened a new log
I20260927 22:02:05.904868 11938 ts_tablet_manager.cc:1397] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260927 22:02:05.905807 11938 raft_consensus.cc:348] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.906100 11938 raft_consensus.cc:374] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.906183 11938 raft_consensus.cc:729] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63a16926ad454239b4914139056e2a2a, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.906481 11939 ts_tablet_manager.cc:1397] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.019s
I20260927 22:02:05.906507 11938 consensus_queue.cc:260] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.907081 11938 ts_tablet_manager.cc:1428] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 22:02:05.907826 11939 raft_consensus.cc:348] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.908113 11939 raft_consensus.cc:374] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:05.908275 11939 raft_consensus.cc:729] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6b55eeb6d74baa97e4f23497e03b01, State: Initialized, Role: FOLLOWER
I20260927 22:02:05.908572 11939 consensus_queue.cc:260] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.909116 11939 ts_tablet_manager.cc:1428] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260927 22:02:05.919811 11878 raft_consensus.cc:482] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:02:05.920133 11878 raft_consensus.cc:504] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.921033 11878 leader_election.cc:288] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2:33197), 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265)
I20260927 22:02:05.921825 11645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e663c0734c4f72bb886c7539a50236" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" is_pre_election: true
I20260927 22:02:05.922224 11645 raft_consensus.cc:2454] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 0.
I20260927 22:02:05.922137 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e663c0734c4f72bb886c7539a50236" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a16926ad454239b4914139056e2a2a" is_pre_election: true
I20260927 22:02:05.922458 11783 raft_consensus.cc:2454] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 0.
I20260927 22:02:05.922758 11431 leader_election.cc:302] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f6b55eeb6d74baa97e4f23497e03b01, cc9a8f4a2b7f47afb5790bba81fec074; no voters:
I20260927 22:02:05.923178 11878 raft_consensus.cc:2781] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:02:05.923310 11878 raft_consensus.cc:482] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:02:05.923434 11878 raft_consensus.cc:3037] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.926636 11878 raft_consensus.cc:504] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.927471 11878 leader_election.cc:288] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 election: Requested vote from peers 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2:33197), 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265)
I20260927 22:02:05.928141 11645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e663c0734c4f72bb886c7539a50236" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6b55eeb6d74baa97e4f23497e03b01"
I20260927 22:02:05.928169 11783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e663c0734c4f72bb886c7539a50236" candidate_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63a16926ad454239b4914139056e2a2a"
I20260927 22:02:05.928407 11783 raft_consensus.cc:3037] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.928431 11645 raft_consensus.cc:3037] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:05.931721 11783 raft_consensus.cc:2454] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 1.
I20260927 22:02:05.931826 11645 raft_consensus.cc:2454] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc9a8f4a2b7f47afb5790bba81fec074 in term 1.
I20260927 22:02:05.932433 11433 leader_election.cc:302] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63a16926ad454239b4914139056e2a2a, cc9a8f4a2b7f47afb5790bba81fec074; no voters:
I20260927 22:02:05.932924 11878 raft_consensus.cc:2781] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:02:05.933188 11878 raft_consensus.cc:686] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 LEADER]: Becoming Leader. State: Replica: cc9a8f4a2b7f47afb5790bba81fec074, State: Running, Role: LEADER
I20260927 22:02:05.933676 11878 consensus_queue.cc:237] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } }
I20260927 22:02:05.937644 11344 catalog_manager.cc:5495] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1). New cstate: current_term: 1 leader_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc9a8f4a2b7f47afb5790bba81fec074" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 } health_report { overall_health: UNKNOWN } } }
I20260927 22:02:05.980957 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:05.973768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.3.159.1:34009)
0927 22:02:05.974027 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:05.974034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:05.974054 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:02:05.974268 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:05.974274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:05.974364 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:05.974450 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:02:05.975050 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.976250 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.978500 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:05.979238 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:05.979370 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:05.979992 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:02:05.980196 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:02:05.980288 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:05.980504 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 22:02:05.980604 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 22:02:05.995043 11645 raft_consensus.cc:1261] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Refusing update from remote peer cc9a8f4a2b7f47afb5790bba81fec074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:05.996120 11783 raft_consensus.cc:1261] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Refusing update from remote peer cc9a8f4a2b7f47afb5790bba81fec074: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:05.996564 11878 consensus_queue.cc:1035] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6b55eeb6d74baa97e4f23497e03b01" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:05.996834 11866 consensus_queue.cc:1035] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63a16926ad454239b4914139056e2a2a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:06.071898 11344 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:02:06.072211 11344 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:02:06.076234 11344 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1240caa533354dd68190d446cd56c853: Sending DeleteTablet for 3 replicas of tablet 01e663c0734c4f72bb886c7539a50236
I20260927 22:02:06.077366 11626 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e663c0734c4f72bb886c7539a50236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:06 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:49352
I20260927 22:02:06.077440 11481 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e663c0734c4f72bb886c7539a50236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:06 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:53730
I20260927 22:02:06.077742 11761 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e663c0734c4f72bb886c7539a50236 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:06 UTC) from {username='oryx', principal='oryx/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:40824
I20260927 22:02:06.078095 11921 tablet_replica.cc:330] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a: stopping tablet replica
I20260927 22:02:06.078119 11922 tablet_replica.cc:330] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074: stopping tablet replica
I20260927 22:02:06.078296 11921 raft_consensus.cc:2229] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:02:06.078296 11922 raft_consensus.cc:2229] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:02:06.078503 11921 raft_consensus.cc:2258] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:06.078653 11922 raft_consensus.cc:2258] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:06.079766 11921 ts_tablet_manager.cc:1905] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:06.080288 11922 ts_tablet_manager.cc:1905] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:06.082792 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 11415
I20260927 22:02:06.085731 11923 tablet_replica.cc:330] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01: stopping tablet replica
I20260927 22:02:06.085992 11923 raft_consensus.cc:2229] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:02:06.086226 11923 raft_consensus.cc:2258] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:06.089968 11921 ts_tablet_manager.cc:1918] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:02:06.090250 11921 log.cc:1196] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/01e663c0734c4f72bb886c7539a50236
I20260927 22:02:06.090324 11922 ts_tablet_manager.cc:1918] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:02:06.090498 11922 log.cc:1196] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/01e663c0734c4f72bb886c7539a50236
I20260927 22:02:06.090967 11921 ts_tablet_manager.cc:1939] T 01e663c0734c4f72bb886c7539a50236 P 63a16926ad454239b4914139056e2a2a: Deleting consensus metadata
I20260927 22:02:06.090972 11922 ts_tablet_manager.cc:1939] T 01e663c0734c4f72bb886c7539a50236 P cc9a8f4a2b7f47afb5790bba81fec074: Deleting consensus metadata
I20260927 22:02:06.092453 11330 catalog_manager.cc:4841] TS cc9a8f4a2b7f47afb5790bba81fec074 (127.3.159.1:34009): tablet 01e663c0734c4f72bb886c7539a50236 (table test-table [id=2d13e7089ebf40d3be1125d1e7f2b299]) successfully deleted
I20260927 22:02:06.092749 11330 catalog_manager.cc:4841] TS 63a16926ad454239b4914139056e2a2a (127.3.159.3:44265): tablet 01e663c0734c4f72bb886c7539a50236 (table test-table [id=2d13e7089ebf40d3be1125d1e7f2b299]) successfully deleted
I20260927 22:02:06.093020 11923 ts_tablet_manager.cc:1905] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:02:06 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.159.1@KRBTEST.COM
I20260927 22:02:06.100580 11923 ts_tablet_manager.cc:1918] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 22:02:06.100788 11923 log.cc:1196] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/01e663c0734c4f72bb886c7539a50236
I20260927 22:02:06.101343 11923 ts_tablet_manager.cc:1939] T 01e663c0734c4f72bb886c7539a50236 P 9f6b55eeb6d74baa97e4f23497e03b01: Deleting consensus metadata
I20260927 22:02:06.102941 11328 catalog_manager.cc:4841] TS 9f6b55eeb6d74baa97e4f23497e03b01 (127.3.159.2:33197): tablet 01e663c0734c4f72bb886c7539a50236 (table test-table [id=2d13e7089ebf40d3be1125d1e7f2b299]) successfully deleted
I20260927 22:02:06.108052 11850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:06.083848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.3.159.1:34009)
0927 22:02:06.084061 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:06.084070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:06.086595 (+ 2525us) server_negotiation.cc:407] Connection header received
0927 22:02:06.086982 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:06.086989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:06.087100 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:06.087183 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:06.089118 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:06.090376 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:06.092288 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:06.092989 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:06.093096 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:06.099875 (+ 6779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:06.099905 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:06.099940 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:06.100004 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:06.104334 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:06.105662 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:06.105671 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:06.105674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:06.105851 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:06.106365 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:06.106370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:06.106372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:06.106811 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:06.107080 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:06.107518 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 22:02:06.107739 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:02:06.109184 11546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:06.265478 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 11415
I20260927 22:02:06.295410 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 11555
Sep 27 22:02:06 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.159.2@KRBTEST.COM
I20260927 22:02:06.318912 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:06.296484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.3.159.2:33197)
0927 22:02:06.296743 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:06.296751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:06.296777 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:02:06.297002 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:06.297010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:06.297120 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:06.297258 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:06.299159 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:06.300440 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:06.302509 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:06.303326 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:06.303552 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:06.309793 (+ 6241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:06.309819 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:06.309841 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:06.309903 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:06.314659 (+ 4756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:06.315902 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:06.315920 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:06.315924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:06.316162 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:06.316668 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:06.316673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:06.316677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:06.317168 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:06.317559 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:06.317969 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 22:02:06.318232 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:02:06.320129 11686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:06.500357 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 11555
I20260927 22:02:06.525621 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 11695
Sep 27 22:02:06 dist-test-slave-sg4j krb5kdc[11299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.159.3@KRBTEST.COM
I20260927 22:02:06.544390 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:06.526616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.3.159.3:44265)
0927 22:02:06.527219 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:06.527224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:06.527241 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:02:06.527307 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:06.527311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:06.527386 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:06.527518 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:06.528909 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:06.529758 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:06.531027 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:06.531495 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:06.531633 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:06.537241 (+ 5608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:06.537258 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:06.537275 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:06.537323 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:06.540790 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:06.541922 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:06.541930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:06.541933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:06.542092 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:06.542555 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:06.542558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:06.542560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:06.543068 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:06.543329 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:06.543733 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 22:02:06.543998 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":300,"threads_started":1}
I20260927 22:02:06.545271 11826 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:06.654778 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 11695
I20260927 22:02:06.677220 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 11312
I20260927 22:02:06.678679 11384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:06.811887 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 11312
2026-09-27T22:02:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6664 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:02:06 dist-test-slave-sg4j krb5kdc[11965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:02:06 dist-test-slave-sg4j krb5kdc[11965](info): set up 2 sockets
Sep 27 22:02:06 dist-test-slave-sg4j krb5kdc[11965](info): commencing operation
krb5kdc: starting...
W20260927 22:02:07.962832 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:02:08 dist-test-slave-sg4j krb5kdc[11965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:02:08.234102 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:36075
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:36075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:02:08.608605 11978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:08.609021 11978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:08.609223 11978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:08.616760 11978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:02:08.616964 11978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:08.617055 11978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:08.617153 11978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:02:08.617228 11978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:02:08.628306 11978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:36075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:36075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:08.630213 11978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:08.631901 11978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:08.646201 11983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:08.646243 11986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:08.646261 11984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:08.647462 11978 server_base.cc:1034] running on GCE node
I20260927 22:02:08.648519 11978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:08.650566 11978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:08.651875 11978 hybrid_clock.cc:648] HybridClock initialized: now 1790546528651782 us; error 82 us; skew 500 ppm
I20260927 22:02:08.652333 11978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T22:02:08Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1914 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:02:08 dist-test-slave-sg4j krb5kdc[11992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:02:08 dist-test-slave-sg4j krb5kdc[11992](info): set up 2 sockets
Sep 27 22:02:08 dist-test-slave-sg4j krb5kdc[11992](info): commencing operation
krb5kdc: starting...
W20260927 22:02:09.886705 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:02:10 dist-test-slave-sg4j krb5kdc[11992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:02:10.167601 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38627
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:38627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:02:10.569363 12005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:10.569902 12005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:10.570079 12005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:10.578311 12005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:02:10.578519 12005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:10.578619 12005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:10.578739 12005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:02:10.578835 12005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:02:10.590479 12005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:38627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:10.592348 12005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:10.594095 12005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:10.610050 12010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:10.610126 12011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:10.610090 12013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:10.611169 12005 server_base.cc:1034] running on GCE node
I20260927 22:02:10.612162 12005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:10.614058 12005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:10.615548 12005 hybrid_clock.cc:648] HybridClock initialized: now 1790546530615446 us; error 83 us; skew 500 ppm
I20260927 22:02:10.616068 12005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:10 dist-test-slave-sg4j krb5kdc[11992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546530, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:10.620841 12005 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:02:10.623003 12005 webserver.cc:460] Webserver started at http://127.3.159.62:37923/ using document root <none> and password file <none>
I20260927 22:02:10.623931 12005 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:10.624058 12005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:10.624482 12005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:10.628043 12005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "70a48f81fc604515ad83043035914195"
format_stamp: "Formatted at 2026-09-27 22:02:10 on dist-test-slave-sg4j"
server_key: "973fbe51346e005426c37cc5254f05a9"
server_key_iv: "c0de01c4d7531e62d9dab6d1b81dfb38"
server_key_version: "encryptionkey@0"
I20260927 22:02:10.628973 12005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "70a48f81fc604515ad83043035914195"
format_stamp: "Formatted at 2026-09-27 22:02:10 on dist-test-slave-sg4j"
server_key: "973fbe51346e005426c37cc5254f05a9"
server_key_iv: "c0de01c4d7531e62d9dab6d1b81dfb38"
server_key_version: "encryptionkey@0"
I20260927 22:02:10.636159 12005 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.004s
I20260927 22:02:10.641938 12020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:10.643939 12005 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260927 22:02:10.644209 12005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "70a48f81fc604515ad83043035914195"
format_stamp: "Formatted at 2026-09-27 22:02:10 on dist-test-slave-sg4j"
server_key: "973fbe51346e005426c37cc5254f05a9"
server_key_iv: "c0de01c4d7531e62d9dab6d1b81dfb38"
server_key_version: "encryptionkey@0"
I20260927 22:02:10.644451 12005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:10.665823 12005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:10.672972 12005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:10.673488 12005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:10.699771 12005 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:38627
I20260927 22:02:10.699869 12071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:38627 every 8 connection(s)
I20260927 22:02:10.701738 12005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:02:10.706605 12072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:10.708065 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 12005
I20260927 22:02:10.708374 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:02:10.709089 3708 external_mini_cluster.cc:1428] Setting key bd15947b1e442a7e0ce956ef0f652f83
I20260927 22:02:10.721701 12072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195: Bootstrap starting.
I20260927 22:02:10.727031 12072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195: Neither blocks nor log segments found. Creating new log.
Sep 27 22:02:10 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:02:10.728499 12072 log.cc:824] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195: Log is configured to *not* fsync() on all Append() calls
I20260927 22:02:10.731946 12072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195: No bootstrap required, opened a new log
I20260927 22:02:10.739677 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:10.712013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.3.159.62:38627)
0927 22:02:10.713329 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:10.713345 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:10.713468 (+ 123us) server_negotiation.cc:407] Connection header received
0927 22:02:10.715675 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:10.715706 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:10.716237 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:10.716661 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:10.718698 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:10.720415 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:10.722582 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:10.723527 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:10.723742 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:10.730110 (+ 6368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:10.730123 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:10.730149 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:10.730209 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:10.734023 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:10.735066 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:10.735075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:10.735085 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:10.735328 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:10.735907 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:10.735912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:10.735914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:10.736762 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:10.737150 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:10.737715 (+ 565us) server_negotiation.cc:299] Negotiation successful
0927 22:02:10.738266 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":611,"threads_started":1}
I20260927 22:02:10.742471 12072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } }
I20260927 22:02:10.743011 12072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:10.743182 12072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a48f81fc604515ad83043035914195, State: Initialized, Role: FOLLOWER
I20260927 22:02:10.744172 12072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } }
I20260927 22:02:10.744432 12072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:02:10.744612 12072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:02:10.744804 12072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:10.747177 12072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } }
I20260927 22:02:10.747759 12072 leader_election.cc:302] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70a48f81fc604515ad83043035914195; no voters:
I20260927 22:02:10.748652 12072 leader_election.cc:288] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:02:10.749001 12077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:02:10.750049 12077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 1 LEADER]: Becoming Leader. State: Replica: 70a48f81fc604515ad83043035914195, State: Running, Role: LEADER
I20260927 22:02:10.750890 12077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } }
I20260927 22:02:10.752530 12072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:02:10.755314 12079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70a48f81fc604515ad83043035914195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } } }
I20260927 22:02:10.756002 12079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:10.755282 12078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70a48f81fc604515ad83043035914195. Latest consensus state: current_term: 1 leader_uuid: "70a48f81fc604515ad83043035914195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } } }
I20260927 22:02:10.756394 12078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:10.766603 12082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:02:10.773088 12082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:02:10.786051 12082 catalog_manager.cc:1347] Generated new cluster ID: b4349b5fb6b94db682aecce4dcf868c7
I20260927 22:02:10.786278 12082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:02:10.809724 12082 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:02:10.811520 12082 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:02:10.828684 12082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195: Generated new TSK 0
I20260927 22:02:10.830039 12082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:02:10.894119 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38627
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:02:11.385649 12098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:11.386031 12098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:11.386202 12098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:11.394757 12098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:11.394920 12098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:11.395124 12098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:02:11.406821 12098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:11.408641 12098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:11.410418 12098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:11.432284 12106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:11.432639 12104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:11.433249 12103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:11.434019 12098 server_base.cc:1034] running on GCE node
I20260927 22:02:11.435019 12098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:11.436563 12098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:11.437849 12098 hybrid_clock.cc:648] HybridClock initialized: now 1790546531437769 us; error 67 us; skew 500 ppm
I20260927 22:02:11.438387 12098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:11 dist-test-slave-sg4j krb5kdc[11992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:11.443915 12098 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:02:11.446252 12098 webserver.cc:460] Webserver started at http://127.3.159.1:38177/ using document root <none> and password file <none>
I20260927 22:02:11.447232 12098 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:11.447419 12098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:11.447829 12098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:11.451880 12098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "8e8ae24fa77e42ba97f82a46a25466c8"
format_stamp: "Formatted at 2026-09-27 22:02:11 on dist-test-slave-sg4j"
server_key: "a4c061bc7c9e7a3e540e2c6d635be1e4"
server_key_iv: "eb0b8ce485fcaebb12c45e93109f96e2"
server_key_version: "encryptionkey@0"
I20260927 22:02:11.452929 12098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "8e8ae24fa77e42ba97f82a46a25466c8"
format_stamp: "Formatted at 2026-09-27 22:02:11 on dist-test-slave-sg4j"
server_key: "a4c061bc7c9e7a3e540e2c6d635be1e4"
server_key_iv: "eb0b8ce485fcaebb12c45e93109f96e2"
server_key_version: "encryptionkey@0"
I20260927 22:02:11.460739 12098 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 22:02:11.466574 12113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:11.469182 12098 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:02:11.469534 12098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "8e8ae24fa77e42ba97f82a46a25466c8"
format_stamp: "Formatted at 2026-09-27 22:02:11 on dist-test-slave-sg4j"
server_key: "a4c061bc7c9e7a3e540e2c6d635be1e4"
server_key_iv: "eb0b8ce485fcaebb12c45e93109f96e2"
server_key_version: "encryptionkey@0"
I20260927 22:02:11.469861 12098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:11.495499 12098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:11.501082 12098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:11.501560 12098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:11.502842 12098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:11.504624 12098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:11.504739 12098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:11.504910 12098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:11.504994 12098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:11.550925 12098 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:39425
I20260927 22:02:11.550992 12225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:39425 every 8 connection(s)
I20260927 22:02:11.552693 12098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:02:11.554015 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 12098
I20260927 22:02:11.554292 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:02:11.555171 3708 external_mini_cluster.cc:1428] Setting key 8eea4b9656b450147e2406474971cbce
Sep 27 22:02:11 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546531, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:02:11.581446 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:11.556762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:43515 (local address 127.3.159.62:38627)
0927 22:02:11.557370 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:11.557411 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:11.558626 (+ 1215us) server_negotiation.cc:407] Connection header received
0927 22:02:11.560684 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:11.560694 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:11.560844 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:11.561039 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:11.563124 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:11.564463 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:11.566626 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:11.567306 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:11.567484 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:11.572577 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:11.572589 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:11.572598 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:11.572662 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:11.577075 (+ 4413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:11.578294 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:11.578302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:11.578309 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:11.578494 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:11.579035 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:11.579043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:11.579049 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:11.579520 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:11.579693 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:11.580584 (+ 891us) server_negotiation.cc:299] Negotiation successful
0927 22:02:11.580908 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":290,"threads_started":1}
I20260927 22:02:11.582228 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:11.557440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:43515)
0927 22:02:11.558442 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:11.558491 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:11.560340 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:11.561335 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:11.561345 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:11.562110 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:11.562885 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:11.562905 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:11.564771 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:11.564776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:11.566370 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:11.566400 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:11.567693 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:11.567702 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:11.568117 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:11.569144 (+ 1027us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:11.569190 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:11.572187 (+ 2997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:11.577333 (+ 5146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:11.577343 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:11.577360 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:11.578099 (+ 739us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:11.578650 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:11.578654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:11.578657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:11.578879 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:11.579678 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:11.579685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:11.580108 (+ 423us) client_negotiation.cc:769] Sending connection context
0927 22:02:11.580393 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 22:02:11.580874 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":409,"threads_started":1}
I20260927 22:02:11.584625 12226 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38627
I20260927 22:02:11.585081 12226 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:02:11.586337 12226 heartbeater.cc:511] Master 127.3.159.62:38627 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:02:11.589337 12037 ts_manager.cc:194] Registered new tserver with Master: 8e8ae24fa77e42ba97f82a46a25466c8 (127.3.159.1:39425)
I20260927 22:02:11.591182 12037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:43515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:02:11.622061 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38627
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:02:12.044509 12232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:12.044903 12232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:12.045017 12232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:12.053766 12232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:12.053934 12232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:12.054075 12232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:02:12.066619 12232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:12.068532 12232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:12.070535 12232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:12.089197 12237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:12.089129 12240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:12.089160 12238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:12.090232 12232 server_base.cc:1034] running on GCE node
I20260927 22:02:12.091013 12232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:12.092825 12232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:12.094210 12232 hybrid_clock.cc:648] HybridClock initialized: now 1790546532094115 us; error 89 us; skew 500 ppm
I20260927 22:02:12.094817 12232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:12 dist-test-slave-sg4j krb5kdc[11992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:12.099349 12232 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:02:12.101650 12232 webserver.cc:460] Webserver started at http://127.3.159.2:45283/ using document root <none> and password file <none>
I20260927 22:02:12.102691 12232 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:12.102828 12232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:12.103188 12232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:12.107231 12232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "5f5b121201ae4a50a056711846e65135"
format_stamp: "Formatted at 2026-09-27 22:02:12 on dist-test-slave-sg4j"
server_key: "2fa6f342816e52fdb2a534f40f60e7bd"
server_key_iv: "b44eb4c69439dfdbeb6711bcfab3b3eb"
server_key_version: "encryptionkey@0"
I20260927 22:02:12.108261 12232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "5f5b121201ae4a50a056711846e65135"
format_stamp: "Formatted at 2026-09-27 22:02:12 on dist-test-slave-sg4j"
server_key: "2fa6f342816e52fdb2a534f40f60e7bd"
server_key_iv: "b44eb4c69439dfdbeb6711bcfab3b3eb"
server_key_version: "encryptionkey@0"
I20260927 22:02:12.116122 12232 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 22:02:12.121508 12247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:12.123927 12232 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:02:12.124176 12232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "5f5b121201ae4a50a056711846e65135"
format_stamp: "Formatted at 2026-09-27 22:02:12 on dist-test-slave-sg4j"
server_key: "2fa6f342816e52fdb2a534f40f60e7bd"
server_key_iv: "b44eb4c69439dfdbeb6711bcfab3b3eb"
server_key_version: "encryptionkey@0"
I20260927 22:02:12.124445 12232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:12.170074 12232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:12.177652 12232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:12.178021 12232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:12.179296 12232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:12.181253 12232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:12.181350 12232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:12.181564 12232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:12.181625 12232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:12.226792 12232 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:41683
I20260927 22:02:12.226871 12359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:41683 every 8 connection(s)
I20260927 22:02:12.228586 12232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:02:12.235474 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 12232
I20260927 22:02:12.235762 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:02:12.236305 3708 external_mini_cluster.cc:1428] Setting key 058cd968ab4478d7988f1ede254acd97
Sep 27 22:02:12 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:02:12.258297 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:12.231969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:42773 (local address 127.3.159.62:38627)
0927 22:02:12.232620 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:12.232626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:12.234266 (+ 1640us) server_negotiation.cc:407] Connection header received
0927 22:02:12.236920 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:12.236929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:12.237065 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:12.237189 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:12.239859 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:12.241185 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:12.243292 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:12.244134 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:12.244325 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:12.249494 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:12.249501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:12.249505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:12.249553 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:12.253173 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:12.254796 (+ 1623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:12.254802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:12.254805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:12.255005 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:12.255749 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:12.255754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:12.255756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:12.256152 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:12.256368 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:12.257633 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0927 22:02:12.257904 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":362,"threads_started":1}
I20260927 22:02:12.259877 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:12.232806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:42773)
0927 22:02:12.233956 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:12.234020 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:12.236453 (+ 2433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:12.237676 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:12.237688 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:12.238426 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:12.239579 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:12.239610 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:12.241556 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:12.241562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:12.243025 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:12.243044 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:12.244469 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:12.244475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:12.244747 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:12.245742 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:12.245775 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:12.248978 (+ 3203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:12.253478 (+ 4500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:12.253501 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:12.253552 (+ 51us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:12.254439 (+ 887us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:12.255180 (+ 741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:12.255188 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:12.255195 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:12.255533 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:12.256385 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:12.256398 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:12.256963 (+ 565us) client_negotiation.cc:769] Sending connection context
0927 22:02:12.257369 (+ 406us) client_negotiation.cc:240] Negotiation successful
0927 22:02:12.258124 (+ 755us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":548,"threads_started":1}
I20260927 22:02:12.262472 12360 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38627
I20260927 22:02:12.263089 12360 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:12.264739 12360 heartbeater.cc:511] Master 127.3.159.62:38627 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
I20260927 22:02:12.267948 12037 ts_manager.cc:194] Registered new tserver with Master: 5f5b121201ae4a50a056711846e65135 (127.3.159.2:41683)
I20260927 22:02:12.268985 12037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:42773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:02:12.302229 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38627
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:02:12.595916 12226 heartbeater.cc:503] Master 127.3.159.62:38627 was elected leader, sending a full tablet report...
W20260927 22:02:12.696834 12366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:12.697145 12366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:12.697242 12366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:12.708516 12366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:12.708696 12366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:12.708882 12366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:02:12.720078 12366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:12.721908 12366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:12.723531 12366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:12.740700 12371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:12.740721 12372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:12.741804 12374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:12.741809 12366 server_base.cc:1034] running on GCE node
I20260927 22:02:12.742642 12366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:12.743947 12366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:12.745244 12366 hybrid_clock.cc:648] HybridClock initialized: now 1790546532745152 us; error 74 us; skew 500 ppm
I20260927 22:02:12.745757 12366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:12 dist-test-slave-sg4j krb5kdc[11992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:12.749948 12366 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:02:12.751891 12366 webserver.cc:460] Webserver started at http://127.3.159.3:43971/ using document root <none> and password file <none>
I20260927 22:02:12.752823 12366 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:12.752988 12366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:12.753363 12366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:12.757014 12366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "9e0671bade184837960c438069c7dc1b"
format_stamp: "Formatted at 2026-09-27 22:02:12 on dist-test-slave-sg4j"
server_key: "3e9130eef1c03812c38791cb47c9e79b"
server_key_iv: "241d910f1ac4d1d6d6cb5d2687631cda"
server_key_version: "encryptionkey@0"
I20260927 22:02:12.757975 12366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "9e0671bade184837960c438069c7dc1b"
format_stamp: "Formatted at 2026-09-27 22:02:12 on dist-test-slave-sg4j"
server_key: "3e9130eef1c03812c38791cb47c9e79b"
server_key_iv: "241d910f1ac4d1d6d6cb5d2687631cda"
server_key_version: "encryptionkey@0"
I20260927 22:02:12.764348 12366 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 22:02:12.769295 12381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:12.771373 12366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 22:02:12.771663 12366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "9e0671bade184837960c438069c7dc1b"
format_stamp: "Formatted at 2026-09-27 22:02:12 on dist-test-slave-sg4j"
server_key: "3e9130eef1c03812c38791cb47c9e79b"
server_key_iv: "241d910f1ac4d1d6d6cb5d2687631cda"
server_key_version: "encryptionkey@0"
I20260927 22:02:12.771958 12366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:12.811407 12366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:12.816898 12366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:12.817203 12366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:12.818600 12366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:12.820236 12366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:12.820323 12366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:12.820431 12366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:12.820484 12366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:12.863412 12366 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:42995
I20260927 22:02:12.863663 12493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:42995 every 8 connection(s)
I20260927 22:02:12.865099 12366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:02:12.869961 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 12366
I20260927 22:02:12.870193 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:02:12.870795 3708 external_mini_cluster.cc:1428] Setting key 14bb1ac4dbea1238e9adbbe16de3cdb1
Sep 27 22:02:12 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546532, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:02:12.890836 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:12.868430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:46107 (local address 127.3.159.62:38627)
0927 22:02:12.868916 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:12.868921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:12.870590 (+ 1669us) server_negotiation.cc:407] Connection header received
0927 22:02:12.873281 (+ 2691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:12.873288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:12.873460 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:12.873626 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:12.875826 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:12.876716 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:12.878697 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:12.879138 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:12.879283 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:12.883540 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:12.883546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:12.883549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:12.883586 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:12.886962 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:12.888258 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:12.888262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:12.888264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:12.888403 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:12.888896 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:12.888900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:12.888902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:12.889251 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:12.889423 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:12.890310 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 22:02:12.890510 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":297,"threads_started":1}
I20260927 22:02:12.891812 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:12.869086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:46107)
0927 22:02:12.870343 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:12.870416 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:12.872806 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:12.873954 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:12.873965 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:12.874627 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:12.875632 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:12.875659 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:12.876886 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:12.876891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:12.878528 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:12.878543 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:12.879406 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:12.879413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:12.879696 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:12.880542 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:12.880575 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:12.883260 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:12.887236 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:12.887249 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:12.887273 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:12.888054 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:12.888541 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:12.888545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:12.888548 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:12.888758 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:12.889373 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:12.889444 (+ 71us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:12.889867 (+ 423us) client_negotiation.cc:769] Sending connection context
0927 22:02:12.890127 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 22:02:12.890559 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":457,"threads_started":1}
I20260927 22:02:12.893956 12494 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38627
I20260927 22:02:12.894304 12494 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:12.895087 12494 heartbeater.cc:511] Master 127.3.159.62:38627 requested a full tablet report, sending...
I20260927 22:02:12.896921 12037 ts_manager.cc:194] Registered new tserver with Master: 9e0671bade184837960c438069c7dc1b (127.3.159.3:42995)
I20260927 22:02:12.898099 12037 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:46107
I20260927 22:02:12.901172 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab.
Entry for principal oryx/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab.
WARNING: no policy specified for oryx/127.3.159.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab.
Entry for principal oryx/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab.
WARNING: no policy specified for oryx/127.3.159.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.159.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab.
Entry for principal oryx/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab.
I20260927 22:02:13.073271 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 12098
Sep 27 22:02:13 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:02:13.096340 12505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:13.074934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.3.159.1:39425)
0927 22:02:13.075997 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:13.076005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:13.076041 (+ 36us) server_negotiation.cc:407] Connection header received
0927 22:02:13.076117 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:13.076124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:13.076422 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:13.076664 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:13.078567 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:13.079760 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:13.081559 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:13.082229 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:13.082371 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:13.087912 (+ 5541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:13.087947 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:13.087969 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:13.088022 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:13.092434 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:13.093730 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:13.093739 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:13.093741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:13.093960 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:13.094583 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:13.094588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:13.094590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:13.095052 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:13.095355 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:13.095779 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 22:02:13.096029 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":610,"threads_started":1}
I20260927 22:02:13.098061 12221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:13.201050 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 12098
I20260927 22:02:13.218729 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 12232
Sep 27 22:02:13 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:02:13.240675 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:13.219932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.3.159.2:41683)
0927 22:02:13.220679 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:13.220688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:13.220720 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:02:13.220792 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:13.220800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:13.221079 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:13.221251 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:13.223144 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:13.224413 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:13.226538 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:13.227594 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:13.227787 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:13.233933 (+ 6146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:13.233945 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:13.233962 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:13.234020 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:13.237332 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:13.238306 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:13.238315 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:13.238317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:13.238455 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:13.238872 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:13.238876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:13.238879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:13.239285 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:13.239641 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:13.239955 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 22:02:13.240267 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":379,"threads_started":1}
I20260927 22:02:13.242415 12355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:13.335595 12360 heartbeater.cc:503] Master 127.3.159.62:38627 was elected leader, sending a full tablet report...
I20260927 22:02:13.336920 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 12232
W20260927 22:02:13.354344 12021 connection.cc:535] server connection from 127.3.159.2:42773 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 22:02:13.354698 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 12366
Sep 27 22:02:13 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:02:13.375747 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:13.355960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.3.159.3:42995)
0927 22:02:13.356692 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:13.356701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:13.356740 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:02:13.356819 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:13.356830 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:13.357116 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:13.357345 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:13.358870 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:13.359975 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:13.361366 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:13.362160 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:13.362331 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:13.367836 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:13.367857 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:13.367880 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:13.367948 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:13.372132 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:13.373115 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:13.373124 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:13.373132 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:13.373301 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:13.373774 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:13.373792 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:13.373795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:13.374265 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:13.374621 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:13.375004 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 22:02:13.375345 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":354,"threads_started":1}
I20260927 22:02:13.377848 12489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:13.470599 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 12366
I20260927 22:02:13.489349 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 12005
I20260927 22:02:13.491068 12067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:13.602968 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 12005
I20260927 22:02:13.629546 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38627
--webserver_interface=127.3.159.62
--webserver_port=37923
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:38627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:02:14.037003 12512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:14.037460 12512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:14.037566 12512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:14.045948 12512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:02:14.046110 12512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:14.046164 12512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:14.046228 12512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:02:14.046283 12512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:02:14.059239 12512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:38627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=37923
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:14.061000 12512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:14.062696 12512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:14.077809 12517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:14.077673 12520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:14.078639 12518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:14.079324 12512 server_base.cc:1034] running on GCE node
I20260927 22:02:14.080363 12512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:14.082538 12512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:14.083998 12512 hybrid_clock.cc:648] HybridClock initialized: now 1790546534083895 us; error 117 us; skew 500 ppm
I20260927 22:02:14.084503 12512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:14 dist-test-slave-sg4j krb5kdc[11992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546534, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:14.092718 12512 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:02:14.094975 12512 webserver.cc:460] Webserver started at http://127.3.159.62:37923/ using document root <none> and password file <none>
I20260927 22:02:14.096052 12512 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:14.096271 12512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:14.104060 12512 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:02:14.108203 12527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:14.110725 12512 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 22:02:14.111059 12512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "70a48f81fc604515ad83043035914195"
format_stamp: "Formatted at 2026-09-27 22:02:10 on dist-test-slave-sg4j"
server_key: "973fbe51346e005426c37cc5254f05a9"
server_key_iv: "c0de01c4d7531e62d9dab6d1b81dfb38"
server_key_version: "encryptionkey@0"
I20260927 22:02:14.113098 12512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:14.131528 12512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:14.136879 12512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:14.137225 12512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:14.162870 12512 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:38627
I20260927 22:02:14.162937 12578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:38627 every 8 connection(s)
I20260927 22:02:14.165133 12512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:02:14.170951 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 12512
I20260927 22:02:14.171851 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:39425
--local_ip_for_outbound_sockets=127.3.159.1
--tserver_master_addrs=127.3.159.62:38627
--webserver_port=38177
--webserver_interface=127.3.159.1
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.159.1
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:02:14.174921 12579 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:02:14.179198 12579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195: Bootstrap starting.
I20260927 22:02:14.214361 12579 log.cc:824] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195: Log is configured to *not* fsync() on all Append() calls
I20260927 22:02:14.229696 12579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:02:14.230572 12579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195: Bootstrap complete.
I20260927 22:02:14.247145 12579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } }
I20260927 22:02:14.248843 12579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70a48f81fc604515ad83043035914195, State: Initialized, Role: FOLLOWER
I20260927 22:02:14.250041 12579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } }
I20260927 22:02:14.250456 12579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:02:14.250612 12579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:02:14.250835 12579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:02:14.259013 12579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } }
I20260927 22:02:14.260267 12579 leader_election.cc:302] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70a48f81fc604515ad83043035914195; no voters:
I20260927 22:02:14.261459 12579 leader_election.cc:288] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:02:14.261708 12583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:02:14.263518 12583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [term 2 LEADER]: Becoming Leader. State: Replica: 70a48f81fc604515ad83043035914195, State: Running, Role: LEADER
I20260927 22:02:14.264454 12583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } }
I20260927 22:02:14.264907 12579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:02:14.268051 12584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "70a48f81fc604515ad83043035914195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } } }
I20260927 22:02:14.268064 12585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70a48f81fc604515ad83043035914195. Latest consensus state: current_term: 2 leader_uuid: "70a48f81fc604515ad83043035914195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70a48f81fc604515ad83043035914195" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38627 } } }
I20260927 22:02:14.268575 12584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:14.268576 12585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195 [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:14.280022 12589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:02:14.287308 12589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:02:14.289047 12589 catalog_manager.cc:1259] Loaded cluster ID: b4349b5fb6b94db682aecce4dcf868c7
I20260927 22:02:14.289438 12589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:02:14.293994 12589 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:02:14.295876 12589 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 70a48f81fc604515ad83043035914195: Loaded TSK: 0
I20260927 22:02:14.297883 12589 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:02:14.645046 12581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:14.645553 12581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:14.645727 12581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:14.654551 12581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:14.654722 12581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:14.654938 12581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:02:14.667035 12581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.1.keytab
--principal=oryx/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:39425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=38177
--tserver_master_addrs=127.3.159.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:14.668980 12581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:14.670836 12581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:14.691421 12606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:14.691408 12608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:14.692485 12605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:14.692417 12581 server_base.cc:1034] running on GCE node
I20260927 22:02:14.693545 12581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:14.694859 12581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:14.696126 12581 hybrid_clock.cc:648] HybridClock initialized: now 1790546534696075 us; error 56 us; skew 500 ppm
I20260927 22:02:14.696573 12581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:14 dist-test-slave-sg4j krb5kdc[11992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546534, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:14.700835 12581 init.cc:377] Logged in from keytab as oryx/127.3.159.1@KRBTEST.COM (short username oryx)
I20260927 22:02:14.702980 12581 webserver.cc:460] Webserver started at http://127.3.159.1:38177/ using document root <none> and password file <none>
I20260927 22:02:14.703948 12581 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:14.704140 12581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:14.711536 12581 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:02:14.715514 12615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:14.717885 12581 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:02:14.718160 12581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "8e8ae24fa77e42ba97f82a46a25466c8"
format_stamp: "Formatted at 2026-09-27 22:02:11 on dist-test-slave-sg4j"
server_key: "a4c061bc7c9e7a3e540e2c6d635be1e4"
server_key_iv: "eb0b8ce485fcaebb12c45e93109f96e2"
server_key_version: "encryptionkey@0"
I20260927 22:02:14.719463 12581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:14.746713 12581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:14.753125 12581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:14.753575 12581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:14.754987 12581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:14.756875 12581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:14.757028 12581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:14.757195 12581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:14.757294 12581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:14.801306 12581 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:39425
I20260927 22:02:14.801471 12727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:39425 every 8 connection(s)
I20260927 22:02:14.803246 12581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:02:14.812434 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 12581
I20260927 22:02:14.813068 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:41683
--local_ip_for_outbound_sockets=127.3.159.2
--tserver_master_addrs=127.3.159.62:38627
--webserver_port=45283
--webserver_interface=127.3.159.2
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.159.2
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:02:14 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:14 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:14.829041 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:14.806817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:57477)
0927 22:02:14.807852 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:14.807907 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:14.810019 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:14.811847 (+ 1828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:14.811858 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:14.812710 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:14.813693 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:14.813723 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.816379 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.816384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:14.818115 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:14.818132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.820335 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.820356 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:14.820592 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:14.821522 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:14.821601 (+ 79us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:14.827599 (+ 5998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":431,"threads_started":1}
W20260927 22:02:14.829022 12731 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:14.806927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:57477 (local address 127.3.159.62:38627)
0927 22:02:14.807779 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:14.807796 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:14.808083 (+ 287us) server_negotiation.cc:407] Connection header received
0927 22:02:14.810745 (+ 2662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:14.810762 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:14.811160 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:14.811575 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:14.814005 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.816104 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.818476 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.819792 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.820307 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:14.827940 (+ 7633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:57477: BlockingRecv error: recv got EOF from 127.3.159.1:57477 (error 108)
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":360,"threads_started":1}
W20260927 22:02:14.830199 12728 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:38627 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.159.62:38627: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Sep 27 22:02:14 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:14 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:14.840951 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:14.830810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:47645)
0927 22:02:14.831012 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:14.831037 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:14.831138 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:14.831783 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:14.831788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:14.832173 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:14.832324 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:14.832332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.833348 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.833354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:14.834324 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:14.834335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.835040 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.835047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:14.835222 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:14.835867 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:14.835885 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:14.840483 (+ 4598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 22:02:14.840971 12731 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:14.831197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:47645 (local address 127.3.159.62:38627)
0927 22:02:14.831460 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:14.831464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:14.831483 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:02:14.831535 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:14.831540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:14.831617 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:14.831753 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:14.832470 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.833194 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.834468 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.834884 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.835022 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:14.840724 (+ 5702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:47645: BlockingRecv error: recv got EOF from 127.3.159.1:47645 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 27 22:02:14 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:14 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:14.853786 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:14.842218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:55317)
0927 22:02:14.842620 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:14.842644 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:14.842801 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:14.843184 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:14.843190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:14.843715 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:14.843973 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:14.843987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.845156 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.845162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:14.846818 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:14.846837 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.847887 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.847894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:14.848129 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:14.848944 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:14.848969 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:14.853432 (+ 4463us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202}
W20260927 22:02:14.854067 12731 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:14.842644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:55317 (local address 127.3.159.62:38627)
0927 22:02:14.842826 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:14.842831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:14.842854 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:02:14.842923 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:14.842927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:14.842990 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:14.843098 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:14.844201 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.844979 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.847075 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:14.847682 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:14.847813 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:14.853669 (+ 5856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:55317: BlockingRecv error: recv got EOF from 127.3.159.1:55317 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 22:02:14.854348 12728 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:02:15.254088 12732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:15.254511 12732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:15.254627 12732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:15.263499 12732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:15.263681 12732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:15.263818 12732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:02:15.276961 12732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.2.keytab
--principal=oryx/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:41683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=45283
--tserver_master_addrs=127.3.159.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:15.278923 12732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:15.280779 12732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:15.299515 12740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:15.299611 12738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:15.299484 12737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:15.303709 12732 server_base.cc:1034] running on GCE node
I20260927 22:02:15.304718 12732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:15.306203 12732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:15.307495 12732 hybrid_clock.cc:648] HybridClock initialized: now 1790546535307402 us; error 80 us; skew 500 ppm
I20260927 22:02:15.307957 12732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546535, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:15.312414 12732 init.cc:377] Logged in from keytab as oryx/127.3.159.2@KRBTEST.COM (short username oryx)
I20260927 22:02:15.314810 12732 webserver.cc:460] Webserver started at http://127.3.159.2:45283/ using document root <none> and password file <none>
I20260927 22:02:15.315793 12732 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:15.315945 12732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:15.323479 12732 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 22:02:15.327520 12747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:15.329703 12732 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:02:15.329923 12732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "5f5b121201ae4a50a056711846e65135"
format_stamp: "Formatted at 2026-09-27 22:02:12 on dist-test-slave-sg4j"
server_key: "2fa6f342816e52fdb2a534f40f60e7bd"
server_key_iv: "b44eb4c69439dfdbeb6711bcfab3b3eb"
server_key_version: "encryptionkey@0"
I20260927 22:02:15.331136 12732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:15.361272 12732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:15.368216 12732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:15.368599 12732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:15.370021 12732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:15.371795 12732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:15.371908 12732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:15.371996 12732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:15.372059 12732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:15.414362 12732 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:41683
I20260927 22:02:15.414485 12859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:41683 every 8 connection(s)
I20260927 22:02:15.416286 12732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:02:15.418004 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 12732
I20260927 22:02:15.418728 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:42995
--local_ip_for_outbound_sockets=127.3.159.3
--tserver_master_addrs=127.3.159.62:38627
--webserver_port=43971
--webserver_interface=127.3.159.3
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.159.3
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:02:15.443790 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:15.426504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:51681 (local address 127.3.159.62:38627)
0927 22:02:15.427120 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:15.427131 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:15.428350 (+ 1219us) server_negotiation.cc:407] Connection header received
0927 22:02:15.430853 (+ 2503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:15.430859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:15.430971 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:15.431124 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:15.433095 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.433976 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.435921 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.436504 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.436655 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:15.443417 (+ 6762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:51681: BlockingRecv error: recv got EOF from 127.3.159.2:51681 (error 108)
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":364,"threads_started":1}
I20260927 22:02:15.444739 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:15.427034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:51681)
0927 22:02:15.428109 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:15.428166 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:15.430480 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:15.431437 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:15.431448 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:15.432111 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:15.432879 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:15.432901 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.434165 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.434170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:15.435726 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:15.435743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.436796 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.436804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:15.437099 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:15.438063 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:15.438092 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:15.443114 (+ 5022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":421,"threads_started":1}
W20260927 22:02:15.445849 12860 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:38627 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.159.62:38627: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:15.456991 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:15.446824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:44271)
0927 22:02:15.447076 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:15.447110 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:15.447304 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:15.447756 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:15.447760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:15.448153 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:15.448404 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:15.448415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.450045 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.450050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:15.451250 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:15.451270 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.452386 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.452391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:15.452575 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:15.453478 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:15.453503 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:15.456743 (+ 3240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 22:02:15.457355 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:15.446991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:44271 (local address 127.3.159.62:38627)
0927 22:02:15.447264 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:15.447272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:15.447302 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:02:15.447474 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:15.447480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:15.447576 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:15.447713 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:15.448566 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.449830 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.451443 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.452145 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.452293 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:15.457076 (+ 4783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:44271: BlockingRecv error: recv got EOF from 127.3.159.2:44271 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:15.468868 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:15.458091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:49459)
0927 22:02:15.458398 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:15.458417 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:15.458535 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:15.458902 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:15.458905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:15.459328 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:15.459531 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:15.459541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.461304 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.461309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:15.462481 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:15.462498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.463563 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.463569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:15.463749 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:15.464551 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:15.464571 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:15.468580 (+ 4009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134}
W20260927 22:02:15.469339 12860 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:02:15.469555 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:15.458263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:49459 (local address 127.3.159.62:38627)
0927 22:02:15.458511 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:15.458517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:15.458541 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:02:15.458651 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:15.458656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:15.458737 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:15.458851 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:15.459915 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.461056 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.462680 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.463354 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.463513 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:15.468913 (+ 5400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:49459: BlockingRecv error: recv got EOF from 127.3.159.2:49459 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260927 22:02:15.820328 12862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:15.820737 12862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:15.820847 12862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:15.830515 12862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:15.830786 12862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:15.830988 12862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:02:15.843582 12862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/krb5kdc/oryx_127.3.159.3.keytab
--principal=oryx/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:42995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=43971
--tserver_master_addrs=127.3.159.62:38627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:15.845863 12862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:15.847831 12862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:02:15.867888 12870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:15.868431 12871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:15.869117 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:15.855218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:42945)
0927 22:02:15.856007 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:15.856035 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:15.856244 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:15.856798 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:15.856804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:15.857548 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:15.857833 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:15.857848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.859612 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.859620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:15.860710 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:15.860726 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.862303 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.862309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:15.862460 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:15.863461 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:15.863483 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:15.868710 (+ 5227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":381,"threads_started":1}
W20260927 22:02:15.869138 12873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:15.869316 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:15.855303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:42945 (local address 127.3.159.62:38627)
0927 22:02:15.855617 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:15.855626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:15.856185 (+ 559us) server_negotiation.cc:407] Connection header received
0927 22:02:15.856416 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:15.856422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:15.856522 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:15.856670 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:15.858128 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.859369 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.860951 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.861956 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.862222 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:15.869014 (+ 6792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:42945: BlockingRecv error: recv got EOF from 127.3.159.1:42945 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 22:02:15.871239 12862 server_base.cc:1034] running on GCE node
I20260927 22:02:15.872282 12862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:15.874125 12862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:15.875710 12862 hybrid_clock.cc:648] HybridClock initialized: now 1790546535875608 us; error 88 us; skew 500 ppm
I20260927 22:02:15.876206 12862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546535, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:15.880406 12862 init.cc:377] Logged in from keytab as oryx/127.3.159.3@KRBTEST.COM (short username oryx)
I20260927 22:02:15.882686 12862 webserver.cc:460] Webserver started at http://127.3.159.3:43971/ using document root <none> and password file <none>
I20260927 22:02:15.883556 12862 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:15.883694 12862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:15.890960 12862 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:02:15.895085 12880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:15.897073 12862 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260927 22:02:15.897341 12862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "9e0671bade184837960c438069c7dc1b"
format_stamp: "Formatted at 2026-09-27 22:02:12 on dist-test-slave-sg4j"
server_key: "3e9130eef1c03812c38791cb47c9e79b"
server_key_iv: "241d910f1ac4d1d6d6cb5d2687631cda"
server_key_version: "encryptionkey@0"
I20260927 22:02:15.898834 12862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:15.921202 12862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:15.927830 12862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:15.928210 12862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:15.929736 12862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:15.931833 12862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:15.931955 12862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:15.932098 12862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:15.932178 12862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:15.975554 12862 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:42995
I20260927 22:02:15.975674 12992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:42995 every 8 connection(s)
I20260927 22:02:15.977523 12862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:02:15.978843 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 12862
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:15 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 22:02:16.000515 12996 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:15.981096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:32951 (local address 127.3.159.62:38627)
0927 22:02:15.981679 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:15.981685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:15.983047 (+ 1362us) server_negotiation.cc:407] Connection header received
0927 22:02:15.985947 (+ 2900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:15.985953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:15.986047 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:15.986194 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:15.988567 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.989660 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.991869 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.992471 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.992624 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:16.000120 (+ 7496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:32951: BlockingRecv error: recv got EOF from 127.3.159.3:32951 (error 108)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":710,"threads_started":1}
I20260927 22:02:16.000789 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:15.980229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47740 (local address 127.3.159.62:38627)
0927 22:02:15.980497 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:15.980504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:15.980872 (+ 368us) server_negotiation.cc:407] Connection header received
0927 22:02:15.981406 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:15.981415 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:15.981526 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:15.981683 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:15.983835 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.985093 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.987168 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.987839 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.987994 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:15.991013 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:15.991023 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:15.991048 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:15.991107 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:15.995756 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:15.996822 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:15.996831 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:15.996840 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:15.997126 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:15.997899 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:15.997907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:15.997913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:15.998752 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:15.999145 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:15.999831 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 22:02:16.000317 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 22:02:16.001426 12997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:15.981784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:32951)
0927 22:02:15.982856 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:15.982913 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:15.985601 (+ 2688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:15.986557 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:15.986570 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:15.987362 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:15.988364 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:15.988390 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.989846 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.989851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:15.991601 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:15.991620 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:15.992781 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:15.992788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:15.993192 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:15.993964 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:15.994000 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:15.999695 (+ 5695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":481,"threads_started":1}
W20260927 22:02:16.002441 12993 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:38627 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.159.62:38627: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Sep 27 22:02:16 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:16 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:16.013520 12997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:16.003221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:47589)
0927 22:02:16.003458 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:16.003481 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:16.003729 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:16.004154 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:16.004157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:16.004666 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:16.004850 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:16.004860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.006143 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.006148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:16.007146 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:16.007159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.008291 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.008297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:16.008462 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:16.009217 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:16.009236 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:16.013165 (+ 3929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 22:02:16.013948 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:16.003267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:47589 (local address 127.3.159.62:38627)
0927 22:02:16.003555 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:16.003560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:16.003580 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:02:16.003892 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:16.003896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:16.003967 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:16.004133 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:16.005029 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.005956 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.007334 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.008056 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.008221 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:16.013615 (+ 5394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:47589: BlockingRecv error: recv got EOF from 127.3.159.3:47589 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Sep 27 22:02:16 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:16 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:16.023857 12997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:16.014418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:49813)
0927 22:02:16.014627 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:16.014656 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:16.014814 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:16.015193 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:16.015198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:16.015577 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:16.015759 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:16.015770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.016857 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.016865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:16.018101 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:16.018117 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.018984 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.018990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:16.019239 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:16.019906 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:16.019925 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:16.023524 (+ 3599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 22:02:16.024260 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:16.014621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:49813 (local address 127.3.159.62:38627)
0927 22:02:16.014861 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:16.014870 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:16.014895 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:02:16.014974 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:16.014978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:16.015052 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:16.015161 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:16.015917 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.016711 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.018256 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.018820 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.018966 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:16.023881 (+ 4915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:49813: BlockingRecv error: recv got EOF from 127.3.159.3:49813 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 22:02:16.024302 12993 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 22:02:16 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:16 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:16.480263 12998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:16.470149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:37445)
0927 22:02:16.470864 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:16.470883 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:16.471024 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:16.471367 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:16.471370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:16.471926 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:16.472155 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:16.472166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.473317 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.473321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:16.474447 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:16.474460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.475395 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.475399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:16.475530 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:16.476376 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:16.476395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:16.479941 (+ 3546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":352,"threads_started":1}
W20260927 22:02:16.480633 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:16.470293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:37445 (local address 127.3.159.62:38627)
0927 22:02:16.470522 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:16.470529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:16.470987 (+ 458us) server_negotiation.cc:407] Connection header received
0927 22:02:16.471125 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:16.471129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:16.471213 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:16.471315 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:16.472354 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.473158 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.474614 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.475192 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.475317 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:16.480251 (+ 4934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:37445: BlockingRecv error: recv got EOF from 127.3.159.2:37445 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 22:02:16 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:16 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:16.881809 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:16.870775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:42949)
0927 22:02:16.871641 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:16.871663 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:16.871852 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:16.872302 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:16.872306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:16.872786 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:16.872961 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:16.872971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.874282 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.874287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:16.875423 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:16.875443 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.876585 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.876593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:16.876807 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:16.877867 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:16.877886 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:16.881428 (+ 3542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":368,"threads_started":1}
W20260927 22:02:16.882164 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:16.871053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:42949 (local address 127.3.159.62:38627)
0927 22:02:16.871659 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:16.871667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:16.871817 (+ 150us) server_negotiation.cc:407] Connection header received
0927 22:02:16.871985 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:16.871989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:16.872064 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:16.872195 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:16.873159 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.874062 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.875634 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:16.876390 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:16.876503 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:16.881879 (+ 5376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:42949: BlockingRecv error: recv got EOF from 127.3.159.1:42949 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 27 22:02:17 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:17 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:17.036396 13000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:17.025559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:59819)
0927 22:02:17.026259 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:17.026288 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:17.026463 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:17.026878 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:17.026882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:17.027453 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:17.027718 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:17.027730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.028827 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.028832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:17.030089 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:17.030106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.031010 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.031015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:17.031181 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:17.032082 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:17.032103 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:17.036005 (+ 3902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":390,"threads_started":1}
W20260927 22:02:17.036957 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:17.025633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:59819 (local address 127.3.159.62:38627)
0927 22:02:17.025933 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:17.025937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:17.026435 (+ 498us) server_negotiation.cc:407] Connection header received
0927 22:02:17.026608 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:17.026613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:17.026687 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:17.026805 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:17.027905 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.028649 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.030316 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.030788 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.030911 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:17.036492 (+ 5581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:59819: BlockingRecv error: recv got EOF from 127.3.159.3:59819 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 22:02:17 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:17 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:17.491891 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:17.481735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:59717)
0927 22:02:17.482476 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:17.482496 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:17.482646 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:17.483110 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:17.483114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:17.483589 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:17.483782 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:17.483792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.484894 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.484899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:17.486034 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:17.486048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.486843 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.486848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:17.486966 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:17.487725 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:17.487747 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:17.491459 (+ 3712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":380,"threads_started":1}
W20260927 22:02:17.492120 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:17.481789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:59717 (local address 127.3.159.62:38627)
0927 22:02:17.481978 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:17.481983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:17.482630 (+ 647us) server_negotiation.cc:407] Connection header received
0927 22:02:17.482807 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:17.482811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:17.482900 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:17.483015 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:17.483975 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.484738 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.486202 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.486671 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.486781 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:17.491768 (+ 4987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:59717: BlockingRecv error: recv got EOF from 127.3.159.2:59717 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 22:02:17 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:17 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:17.893060 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:17.882932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:59375)
0927 22:02:17.883581 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:17.883599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:17.883745 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:17.884176 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:17.884181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:17.884627 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:17.884832 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:17.884842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.885990 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.885995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:17.886990 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:17.887003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.887762 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.887767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:17.887930 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:17.888873 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:17.888892 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:17.892628 (+ 3736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":385,"threads_started":1}
W20260927 22:02:17.893137 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:17.883272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:59375 (local address 127.3.159.62:38627)
0927 22:02:17.883494 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:17.883501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:17.883708 (+ 207us) server_negotiation.cc:407] Connection header received
0927 22:02:17.883891 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:17.883895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:17.883972 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:17.884073 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:17.885002 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.885827 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.887152 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:17.887604 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:17.887707 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:17.892870 (+ 5163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:59375: BlockingRecv error: recv got EOF from 127.3.159.1:59375 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 22:02:18 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:18 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:18.048296 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:18.037979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:57985)
0927 22:02:18.038707 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:18.038726 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:18.038917 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:18.039398 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:18.039402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:18.039964 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:18.040219 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:18.040240 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.041313 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.041318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:18.042486 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:18.042504 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.043360 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.043364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:18.043509 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:18.044313 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:18.044334 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:18.047990 (+ 3656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":386,"threads_started":1}
W20260927 22:02:18.048643 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:18.037981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:57985 (local address 127.3.159.62:38627)
0927 22:02:18.038212 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:18.038221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:18.038835 (+ 614us) server_negotiation.cc:407] Connection header received
0927 22:02:18.039067 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:18.039073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:18.039183 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:18.039295 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:18.040395 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.041122 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.042667 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.043146 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.043276 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:18.048309 (+ 5033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:57985: BlockingRecv error: recv got EOF from 127.3.159.3:57985 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 22:02:18 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:18 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:18.504184 13004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:18.493311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:48869)
0927 22:02:18.494091 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:18.494113 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:18.494287 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:18.494699 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:18.494703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:18.495168 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:18.495403 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:18.495414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.496517 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.496522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:18.497777 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:18.497792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.498693 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.498699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:18.498868 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:18.499698 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:18.499718 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:18.503799 (+ 4081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":473,"threads_started":1}
W20260927 22:02:18.504534 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:18.493479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:48869 (local address 127.3.159.62:38627)
0927 22:02:18.493999 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:18.494005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:18.494264 (+ 259us) server_negotiation.cc:407] Connection header received
0927 22:02:18.494424 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:18.494428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:18.494523 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:18.494616 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:18.495597 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.496322 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.497989 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.498491 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.498617 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:18.504157 (+ 5540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:48869: BlockingRecv error: recv got EOF from 127.3.159.2:48869 (error 108)
Metrics: {"server-negotiator.queue_time_us":348}
Sep 27 22:02:18 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:18 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:18.907537 13005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:18.894268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:41341)
0927 22:02:18.894962 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:18.894987 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:18.895185 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:18.895917 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:18.895924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:18.896513 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:18.896713 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:18.896723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.898037 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.898043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:18.899807 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:18.899834 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.901004 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.901014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:18.901262 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:18.902269 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:18.902289 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:18.907159 (+ 4870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":411,"threads_started":1}
W20260927 22:02:18.908185 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:18.894398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:41341 (local address 127.3.159.62:38627)
0927 22:02:18.894644 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:18.894649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:18.895120 (+ 471us) server_negotiation.cc:407] Connection header received
0927 22:02:18.895478 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:18.895484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:18.895580 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:18.895871 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:18.896913 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.897863 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.900134 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:18.900733 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:18.900955 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:18.907517 (+ 6562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:41341: BlockingRecv error: recv got EOF from 127.3.159.1:41341 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 22:02:19 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:19 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:19.058913 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:19.049868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:42209)
0927 22:02:19.050513 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:19.050532 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:19.050728 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:19.051109 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:19.051116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:19.051514 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:19.051698 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:19.051708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.052919 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.052926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:19.053953 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:19.053966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.054735 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.054741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:19.054917 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:19.055599 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:19.055617 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:19.058535 (+ 2918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":355,"threads_started":1}
W20260927 22:02:19.059047 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:19.050049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:42209 (local address 127.3.159.62:38627)
0927 22:02:19.050360 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:19.050367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:19.050649 (+ 282us) server_negotiation.cc:407] Connection header received
0927 22:02:19.050871 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:19.050883 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:19.050949 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:19.051061 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:19.051857 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.052757 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.054117 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.054568 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.054713 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:19.058759 (+ 4046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:42209: BlockingRecv error: recv got EOF from 127.3.159.3:42209 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 27 22:02:19 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:19 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:19.520205 13007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:19.505665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:42139)
0927 22:02:19.506354 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:19.506376 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:19.506664 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:19.507172 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:19.507176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:19.507576 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:19.507758 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:19.507768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.509147 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.509151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:19.510291 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:19.510306 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.511212 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.511217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:19.511357 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:19.516082 (+ 4725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:19.516110 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:19.519932 (+ 3822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":390,"threads_started":1}
W20260927 22:02:19.520627 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:19.505851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:42139 (local address 127.3.159.62:38627)
0927 22:02:19.506150 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:19.506158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:19.506584 (+ 426us) server_negotiation.cc:407] Connection header received
0927 22:02:19.506814 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:19.506820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:19.506926 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:19.507040 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:19.507968 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.508952 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.510476 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.511014 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.511220 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:19.520246 (+ 9026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:42139: BlockingRecv error: recv got EOF from 127.3.159.2:42139 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 27 22:02:19 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:19 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:19.918805 13008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:19.908854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:43527)
0927 22:02:19.909547 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:19.909565 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:19.909756 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:19.910115 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:19.910119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:19.910590 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:19.910756 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:19.910764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.912005 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.912013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:19.912979 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:19.912991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.913795 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.913801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:19.914001 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:19.914762 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:19.914780 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:19.918428 (+ 3648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":317,"threads_started":1}
W20260927 22:02:19.918886 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:19.908945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:43527 (local address 127.3.159.62:38627)
0927 22:02:19.909275 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:19.909282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:19.909660 (+ 378us) server_negotiation.cc:407] Connection header received
0927 22:02:19.909913 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:19.909917 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:19.909982 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:19.910087 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:19.910899 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.911827 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.913143 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:19.913637 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:19.913774 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:19.918657 (+ 4883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:43527: BlockingRecv error: recv got EOF from 127.3.159.1:43527 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Sep 27 22:02:20 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:20 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:20.072472 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:20.060397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:39137)
0927 22:02:20.061058 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:20.061077 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:20.061326 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:20.061770 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:20.061775 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:20.062232 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:20.062401 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:20.062411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.063514 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.063523 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:20.064639 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:20.064659 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.065975 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.065980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:20.066154 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:20.067056 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:20.067076 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:20.071885 (+ 4809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":362,"threads_started":1}
W20260927 22:02:20.072633 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:20.060557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:39137 (local address 127.3.159.62:38627)
0927 22:02:20.060904 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:20.060911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:20.061189 (+ 278us) server_negotiation.cc:407] Connection header received
0927 22:02:20.061542 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:20.061547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:20.061620 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:20.061745 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:20.062554 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.063368 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.064961 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.065787 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.065920 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:20.072183 (+ 6263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:39137: BlockingRecv error: recv got EOF from 127.3.159.3:39137 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 27 22:02:20 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:20 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:20.532279 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:20.521555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:37271)
0927 22:02:20.522423 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:20.522448 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:20.522616 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:20.523014 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:20.523018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:20.523680 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:20.523884 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:20.523895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.525101 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.525105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:20.526254 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:20.526269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.527094 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.527099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:20.527305 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:20.528101 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:20.528123 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:20.531940 (+ 3817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":431,"threads_started":1}
W20260927 22:02:20.532757 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:20.521816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:37271 (local address 127.3.159.62:38627)
0927 22:02:20.522207 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:20.522214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:20.522568 (+ 354us) server_negotiation.cc:407] Connection header received
0927 22:02:20.522739 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:20.522742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:20.522826 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:20.522920 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:20.524065 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.524954 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.526428 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.526905 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.527020 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:20.532271 (+ 5251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:37271: BlockingRecv error: recv got EOF from 127.3.159.2:37271 (error 108)
Metrics: {"server-negotiator.queue_time_us":205}
Sep 27 22:02:20 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:20 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:20.929425 13011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:20.920018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:58201)
0927 22:02:20.920719 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:20.920742 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:20.920898 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:20.921241 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:20.921246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:20.921732 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:20.921880 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:20.921889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.922907 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.922914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:20.924031 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:20.924043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.924760 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.924767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:20.924944 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:20.925735 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:20.925754 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:20.929067 (+ 3313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":356,"threads_started":1}
W20260927 22:02:20.929605 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:20.920115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:58201 (local address 127.3.159.62:38627)
0927 22:02:20.920322 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:20.920330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:20.920817 (+ 487us) server_negotiation.cc:407] Connection header received
0927 22:02:20.921034 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:20.921039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:20.921112 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:20.921214 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:20.922023 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.922737 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.924172 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:20.924586 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:20.924718 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:20.929318 (+ 4600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:58201: BlockingRecv error: recv got EOF from 127.3.159.1:58201 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 22:02:21 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:21 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:21.084730 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:21.073917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:51839)
0927 22:02:21.074701 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:21.074722 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:21.074946 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:21.075419 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:21.075425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:21.076049 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:21.076259 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:21.076270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.077482 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.077492 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:21.078557 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:21.078570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.079381 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.079387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:21.079602 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:21.080341 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:21.080360 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:21.084324 (+ 3964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":417,"threads_started":1}
W20260927 22:02:21.084821 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:21.074043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:51839 (local address 127.3.159.62:38627)
0927 22:02:21.074354 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:21.074365 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:21.074884 (+ 519us) server_negotiation.cc:407] Connection header received
0927 22:02:21.075115 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:21.075120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:21.075198 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:21.075387 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:21.076424 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.077289 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.078722 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.079195 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.079346 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:21.084589 (+ 5243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:51839: BlockingRecv error: recv got EOF from 127.3.159.3:51839 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 27 22:02:21 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:21 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:21.543649 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:21.533729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:35481)
0927 22:02:21.534402 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:21.534422 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:21.534597 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:21.535105 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:21.535111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:21.535745 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:21.535979 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:21.535990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.537275 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.537280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:21.538300 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:21.538314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.539165 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.539172 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:21.539313 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:21.540037 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:21.540057 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:21.543380 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":351,"threads_started":1}
W20260927 22:02:21.543926 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:21.533872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:35481 (local address 127.3.159.62:38627)
0927 22:02:21.534130 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:21.534139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:21.534509 (+ 370us) server_negotiation.cc:407] Connection header received
0927 22:02:21.534767 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:21.534772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:21.534894 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:21.535012 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:21.536163 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.537108 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.538477 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.538977 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.539117 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:21.543651 (+ 4534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:35481: BlockingRecv error: recv got EOF from 127.3.159.2:35481 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 22:02:21 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:21 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:21.942118 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:21.930986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:51489)
0927 22:02:21.931622 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:21.931641 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:21.931780 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:21.932299 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:21.932303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:21.932709 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:21.932890 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:21.932900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.934470 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.934476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:21.935683 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:21.935701 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.936598 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.936606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:21.936968 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:21.937751 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:21.937770 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:21.941386 (+ 3616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":296,"threads_started":1}
W20260927 22:02:21.942319 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:21.931162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:51489 (local address 127.3.159.62:38627)
0927 22:02:21.931413 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:21.931418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:21.931766 (+ 348us) server_negotiation.cc:407] Connection header received
0927 22:02:21.931969 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:21.931974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:21.932053 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:21.932202 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:21.933084 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.934232 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.935872 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:21.936384 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:21.936565 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:21.941992 (+ 5427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:51489: BlockingRecv error: recv got EOF from 127.3.159.1:51489 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 22:02:22 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:22 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:22.099071 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:22.086180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:41995)
0927 22:02:22.087028 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:22.087048 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:22.087318 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:22.087759 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:22.087764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:22.088445 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:22.088714 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:22.088728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.090479 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.090485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:22.091981 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:22.092004 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.094051 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.094057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:22.094248 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:22.095016 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:22.095039 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:22.098776 (+ 3737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":416,"threads_started":1}
W20260927 22:02:22.099646 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:22.086330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:41995 (local address 127.3.159.62:38627)
0927 22:02:22.086684 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:22.086693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:22.087218 (+ 525us) server_negotiation.cc:407] Connection header received
0927 22:02:22.087511 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:22.087517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:22.087598 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:22.087751 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:22.088909 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.090255 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.092233 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.093772 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.093958 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:22.099131 (+ 5173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:41995: BlockingRecv error: recv got EOF from 127.3.159.3:41995 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 27 22:02:22 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:22 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:22.554154 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:22.545000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:33309)
0927 22:02:22.545991 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:22.546009 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:22.546192 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:22.546582 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:22.546586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:22.547045 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:22.547229 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:22.547239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.548236 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.548243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:22.549179 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:22.549190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.549978 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.549984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:22.550154 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:22.550819 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:22.550836 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:22.553744 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":450,"threads_started":1}
W20260927 22:02:22.554215 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:22.545076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:33309 (local address 127.3.159.62:38627)
0927 22:02:22.545510 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:22.545520 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:22.546118 (+ 598us) server_negotiation.cc:407] Connection header received
0927 22:02:22.546339 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:22.546344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:22.546429 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:22.546549 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:22.547382 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.548090 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.549323 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.549807 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.549945 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:22.553970 (+ 4025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:33309: BlockingRecv error: recv got EOF from 127.3.159.2:33309 (error 108)
Metrics: {"server-negotiator.queue_time_us":206}
Sep 27 22:02:22 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:22 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:22.954298 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:22.943557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:55019)
0927 22:02:22.944316 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:22.944346 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:22.944515 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:22.944889 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:22.944893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:22.945506 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:22.945709 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:22.945719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.946881 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.946885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:22.948429 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:22.948448 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.949456 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.949463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:22.949627 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:22.950436 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:22.950456 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:22.953996 (+ 3540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":438,"threads_started":1}
W20260927 22:02:22.954739 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:22.943750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:55019 (local address 127.3.159.62:38627)
0927 22:02:22.944001 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:22.944006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:22.944514 (+ 508us) server_negotiation.cc:407] Connection header received
0927 22:02:22.944632 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:22.944637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:22.944710 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:22.944846 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:22.945897 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.946702 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.948619 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:22.949163 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:22.949292 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:22.954433 (+ 5141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:55019: BlockingRecv error: recv got EOF from 127.3.159.1:55019 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 22:02:23 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:23 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:23.112237 13018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:23.100325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:45871)
0927 22:02:23.101038 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:23.101059 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:23.101228 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:23.101740 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:23.101745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:23.102290 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:23.102730 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:23.102742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.104474 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.104485 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:23.105692 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:23.105714 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.106710 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.106717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:23.106942 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:23.107867 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:23.107895 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:23.111908 (+ 4013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":339,"threads_started":1}
W20260927 22:02:23.112486 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:23.100643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:45871 (local address 127.3.159.62:38627)
0927 22:02:23.100929 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:23.100938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:23.101190 (+ 252us) server_negotiation.cc:407] Connection header received
0927 22:02:23.101361 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:23.101365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:23.101536 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:23.101655 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:23.102984 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.104097 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.105927 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.106483 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.106668 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:23.112173 (+ 5505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:45871: BlockingRecv error: recv got EOF from 127.3.159.3:45871 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 27 22:02:23 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:23 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:23.565647 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:23.555671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:42635)
0927 22:02:23.556392 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:23.556412 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:23.556571 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:23.557001 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:23.557006 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:23.557505 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:23.557726 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:23.557737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.559059 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.559063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:23.560277 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:23.560292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.561160 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.561167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:23.561345 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:23.562163 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:23.562182 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:23.565228 (+ 3046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":388,"threads_started":1}
W20260927 22:02:23.565783 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:23.555866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:42635 (local address 127.3.159.62:38627)
0927 22:02:23.556059 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:23.556064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:23.556581 (+ 517us) server_negotiation.cc:407] Connection header received
0927 22:02:23.556691 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:23.556695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:23.556785 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:23.556921 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:23.557944 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.558877 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.560466 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.560947 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.561121 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:23.565515 (+ 4394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:42635: BlockingRecv error: recv got EOF from 127.3.159.2:42635 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 22:02:23 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:23 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:23.965873 13020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:23.955441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:56797)
0927 22:02:23.956238 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:23.956260 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:23.956480 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:23.956896 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:23.956901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:23.957419 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:23.957654 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:23.957664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.958794 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.958802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:23.959853 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:23.959866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.960683 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.960690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:23.960901 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:23.961740 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:23.961762 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:23.965457 (+ 3695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":406,"threads_started":1}
W20260927 22:02:23.966217 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:23.955668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:56797 (local address 127.3.159.62:38627)
0927 22:02:23.956010 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:23.956018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:23.956385 (+ 367us) server_negotiation.cc:407] Connection header received
0927 22:02:23.956632 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:23.956639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:23.956733 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:23.956855 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:23.957824 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.958632 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.960028 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:23.960493 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:23.960650 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:23.965806 (+ 5156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:56797: BlockingRecv error: recv got EOF from 127.3.159.1:56797 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 27 22:02:24 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:24 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:24.124526 13021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:24.113543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:44499)
0927 22:02:24.114744 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:24.114765 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:24.114973 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:24.115365 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:24.115370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:24.115887 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:24.116066 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:24.116076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.117448 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.117455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:24.118539 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:24.118551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.119470 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.119479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:24.119670 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:24.120376 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:24.120395 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:24.124082 (+ 3687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":993,"thread_start_us":872,"threads_started":1}
W20260927 22:02:24.124770 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:24.113605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:44499 (local address 127.3.159.62:38627)
0927 22:02:24.113867 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:24.113874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:24.114880 (+ 1006us) server_negotiation.cc:407] Connection header received
0927 22:02:24.115125 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:24.115142 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:24.115214 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:24.115335 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:24.116228 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.117260 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.118718 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.119220 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.119396 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:24.124401 (+ 5005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:44499: BlockingRecv error: recv got EOF from 127.3.159.3:44499 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 22:02:24 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:24 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:24.581502 13022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:24.567123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:36097)
0927 22:02:24.567898 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:24.567943 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:24.568108 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:24.568614 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:24.568617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:24.569012 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:24.569200 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:24.569210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.570552 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.570557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:24.571577 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:24.571591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.572531 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.572536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:24.572693 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:24.577089 (+ 4396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:24.577115 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:24.581049 (+ 3934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":500,"threads_started":1}
W20260927 22:02:24.581683 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:24.567191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:36097 (local address 127.3.159.62:38627)
0927 22:02:24.567373 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:24.567377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:24.568056 (+ 679us) server_negotiation.cc:407] Connection header received
0927 22:02:24.568275 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:24.568285 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:24.568399 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:24.568548 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:24.569361 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.570327 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.571744 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.572310 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.572420 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:24.581346 (+ 8926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:36097: BlockingRecv error: recv got EOF from 127.3.159.2:36097 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 22:02:24 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:24 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:24.978138 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:24.967227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:41249)
0927 22:02:24.967842 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:24.967859 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:24.968052 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:24.968405 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:24.968410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:24.968820 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:24.968982 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:24.968991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.970110 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.970117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:24.971066 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:24.971077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.971828 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.971834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:24.972204 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:24.973098 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:24.973133 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:24.977732 (+ 4599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":354,"threads_started":1}
W20260927 22:02:24.978207 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:24.967319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:41249 (local address 127.3.159.62:38627)
0927 22:02:24.967652 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:24.967661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:24.967975 (+ 314us) server_negotiation.cc:407] Connection header received
0927 22:02:24.968186 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:24.968199 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:24.968263 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:24.968374 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:24.969144 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.969969 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.971245 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:24.971652 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:24.971785 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:24.977945 (+ 6160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:41249: BlockingRecv error: recv got EOF from 127.3.159.1:41249 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 27 22:02:25 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:25 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:25.136163 13024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:25.125940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:45815)
0927 22:02:25.126501 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:25.126523 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:25.126684 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:25.127157 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:25.127163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:25.127649 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:25.127850 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:25.127861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.129097 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.129102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:25.130179 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:25.130192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.131080 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.131085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:25.131244 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:25.131973 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:25.131993 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:25.135770 (+ 3777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":415,"threads_started":1}
W20260927 22:02:25.136240 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:25.126290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:45815 (local address 127.3.159.62:38627)
0927 22:02:25.126538 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:25.126544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:25.126618 (+ 74us) server_negotiation.cc:407] Connection header received
0927 22:02:25.126817 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:25.126823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:25.126923 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:25.127038 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:25.128046 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.128917 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.130339 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.130860 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.130971 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:25.135975 (+ 5004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:45815: BlockingRecv error: recv got EOF from 127.3.159.3:45815 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 22:02:25 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:25 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:25.594425 13025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:25.582933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:38299)
0927 22:02:25.583856 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:25.583880 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:25.584063 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:25.585122 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:25.585130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:25.585795 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:25.585988 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:25.586000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.587290 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.587297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:25.588529 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:25.588548 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.589583 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.589589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:25.589748 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:25.590629 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:25.590651 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:25.594133 (+ 3482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":551,"threads_started":1}
W20260927 22:02:25.594839 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:25.583351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:38299 (local address 127.3.159.62:38627)
0927 22:02:25.583658 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:25.583666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:25.584043 (+ 377us) server_negotiation.cc:407] Connection header received
0927 22:02:25.584589 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:25.584596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:25.584709 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:25.584859 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:25.586221 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.587068 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.588783 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.589253 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.589430 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:25.594493 (+ 5063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:38299: BlockingRecv error: recv got EOF from 127.3.159.2:38299 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 27 22:02:25 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:25 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:25.989838 13026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:25.979808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:58105)
0927 22:02:25.980614 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:25.980634 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:25.980805 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:25.981278 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:25.981282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:25.981874 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:25.982094 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:25.982103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.983262 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.983267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:25.984249 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:25.984261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.984974 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.984979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:25.985117 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:25.985945 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:25.985972 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:25.989532 (+ 3560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":473,"threads_started":1}
W20260927 22:02:25.990162 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:25.980191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:58105 (local address 127.3.159.62:38627)
0927 22:02:25.980452 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:25.980458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:25.980755 (+ 297us) server_negotiation.cc:407] Connection header received
0927 22:02:25.980977 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:25.980982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:25.981053 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:25.981179 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:25.982280 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.983056 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.984410 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:25.984801 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:25.984918 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:25.989825 (+ 4907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:58105: BlockingRecv error: recv got EOF from 127.3.159.1:58105 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 27 22:02:26 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:26 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:26.153100 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:26.137588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:54447)
0927 22:02:26.138239 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:26.138260 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:26.138417 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:26.138954 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:26.138958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:26.139537 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:26.139709 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:26.139718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.141150 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.141155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:26.142369 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:26.142387 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.143528 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.143534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:26.143751 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:26.149046 (+ 5295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:26.149069 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:26.152652 (+ 3583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":383,"threads_started":1}
W20260927 22:02:26.153301 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:26.137884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:54447 (local address 127.3.159.62:38627)
0927 22:02:26.138185 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:26.138195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:26.138402 (+ 207us) server_negotiation.cc:407] Connection header received
0927 22:02:26.138626 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:26.138632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:26.138735 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:26.138870 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:26.139942 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.140928 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.142588 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.143192 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.143466 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:26.152949 (+ 9483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:54447: BlockingRecv error: recv got EOF from 127.3.159.3:54447 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 27 22:02:26 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:26 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:26.607239 13028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:26.595928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:57093)
0927 22:02:26.596522 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:26.596541 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:26.596682 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:26.597153 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:26.597158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:26.597642 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:26.597846 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:26.597857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.599250 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.599256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:26.600547 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:26.600572 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.601847 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.601854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:26.602151 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:26.603092 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:26.603160 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:26.606932 (+ 3772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":324,"threads_started":1}
W20260927 22:02:26.607601 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:26.596085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:57093 (local address 127.3.159.62:38627)
0927 22:02:26.596382 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:26.596388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:26.596679 (+ 291us) server_negotiation.cc:407] Connection header received
0927 22:02:26.596822 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:26.596827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:26.596917 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:26.597055 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:26.598078 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.598995 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.600852 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.601584 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.601793 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:26.607259 (+ 5466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:57093: BlockingRecv error: recv got EOF from 127.3.159.2:57093 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
Sep 27 22:02:26 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:26 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:27.001178 13029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:26.991040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:33205)
0927 22:02:26.991825 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:26.991844 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:26.992027 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:26.992402 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:26.992407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:26.992897 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:26.993079 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:26.993088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.994468 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.994475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:26.995484 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:26.995496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.996259 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.996266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:26.996464 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:26.997256 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:26.997276 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:27.000768 (+ 3492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":451,"threads_started":1}
W20260927 22:02:27.001420 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:26.991280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:33205 (local address 127.3.159.62:38627)
0927 22:02:26.991535 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:26.991542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:26.991952 (+ 410us) server_negotiation.cc:407] Connection header received
0927 22:02:26.992189 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:26.992193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:26.992262 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:26.992374 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:26.993263 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.994289 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.995643 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:26.996085 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:26.996236 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:27.001073 (+ 4837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:33205: BlockingRecv error: recv got EOF from 127.3.159.1:33205 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 27 22:02:27 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:27 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:27.166555 13030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:27.154519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:46713)
0927 22:02:27.155254 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:27.155280 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:27.155501 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:27.155969 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:27.155973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:27.156474 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:27.156706 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:27.156716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:27.158003 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:27.158009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:27.159304 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:27.159330 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:27.160768 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:27.160776 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:27.160965 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:27.161904 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:27.161929 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:27.166110 (+ 4181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":432,"threads_started":1}
W20260927 22:02:27.166777 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:27.154703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:46713 (local address 127.3.159.62:38627)
0927 22:02:27.154940 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:27.154949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:27.155407 (+ 458us) server_negotiation.cc:407] Connection header received
0927 22:02:27.155720 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:27.155725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:27.155805 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:27.155916 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:27.156914 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:27.157803 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:27.159624 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:27.160215 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:27.160637 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:27.166457 (+ 5820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:46713: BlockingRecv error: recv got EOF from 127.3.159.3:46713 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 22:02:27 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:27 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:27.617842 13031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:27.608498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:49691)
0927 22:02:27.609228 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:27.609248 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:27.609541 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:27.609915 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:27.609919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:27.610332 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:27.610492 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:27.610500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:27.611730 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:27.611738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:27.612703 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:27.612716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:27.613504 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:27.613511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:27.613702 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:27.614445 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:27.614465 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:27.617476 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":425,"threads_started":1}
W20260927 22:02:27.618327 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:27.608664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:49691 (local address 127.3.159.62:38627)
0927 22:02:27.609033 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:27.609043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:27.609411 (+ 368us) server_negotiation.cc:407] Connection header received
0927 22:02:27.609698 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:27.609708 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:27.609770 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:27.609883 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:27.610643 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:27.611525 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:27.612862 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:27.613307 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:27.613484 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:27.617704 (+ 4220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:49691: BlockingRecv error: recv got EOF from 127.3.159.2:49691 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 27 22:02:28 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:28 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:28.012905 13032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:28.002988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:47655)
0927 22:02:28.003658 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:28.003677 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:28.003829 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:28.004229 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:28.004233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:28.004704 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:28.004945 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:28.004965 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.006163 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.006168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:28.007149 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:28.007160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.007979 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.007985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:28.008167 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:28.008921 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:28.008939 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:28.012507 (+ 3568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":419,"threads_started":1}
W20260927 22:02:28.013057 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:28.003088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:47655 (local address 127.3.159.62:38627)
0927 22:02:28.003313 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:28.003317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:28.003768 (+ 451us) server_negotiation.cc:407] Connection header received
0927 22:02:28.003970 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:28.003976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:28.004072 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:28.004195 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:28.005111 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.006016 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.007300 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.007785 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.007931 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:28.012791 (+ 4860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:47655: BlockingRecv error: recv got EOF from 127.3.159.1:47655 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 22:02:28 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:28 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:28.177845 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:28.168066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:51959)
0927 22:02:28.168841 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:28.168859 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:28.169092 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:28.169519 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:28.169523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:28.170054 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:28.170236 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:28.170245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.171360 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.171367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:28.172315 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:28.172326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.173128 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.173135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:28.173326 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:28.174131 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:28.174150 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:28.177478 (+ 3328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":460,"threads_started":1}
W20260927 22:02:28.178069 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:28.168215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:51959 (local address 127.3.159.62:38627)
0927 22:02:28.168670 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:28.168679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:28.168963 (+ 284us) server_negotiation.cc:407] Connection header received
0927 22:02:28.169241 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:28.169247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:28.169328 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:28.169488 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:28.170393 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.171200 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.172470 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.172949 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.173090 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:28.177781 (+ 4691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:51959: BlockingRecv error: recv got EOF from 127.3.159.3:51959 (error 108)
Metrics: {"server-negotiator.queue_time_us":255}
Sep 27 22:02:28 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:28 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:28.628753 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:28.619187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:37761)
0927 22:02:28.619957 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:28.619975 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:28.620162 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:28.620524 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:28.620528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:28.620949 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:28.621099 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:28.621108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.622192 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.622198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:28.623145 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:28.623157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.624018 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.624026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:28.624190 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:28.624989 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:28.625015 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:28.628419 (+ 3404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":412,"threads_started":1}
W20260927 22:02:28.629212 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:28.619515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:37761 (local address 127.3.159.62:38627)
0927 22:02:28.619791 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:28.619799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:28.620075 (+ 276us) server_negotiation.cc:407] Connection header received
0927 22:02:28.620298 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:28.620304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:28.620379 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:28.620491 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:28.621251 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.622047 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.623301 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:28.623811 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:28.623952 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:28.628874 (+ 4922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:37761: BlockingRecv error: recv got EOF from 127.3.159.2:37761 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 27 22:02:29 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:29 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:29.024658 13035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:29.014499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:58431)
0927 22:02:29.015364 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:29.015388 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:29.015601 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:29.016129 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:29.016135 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:29.016702 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:29.016984 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:29.016996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.018396 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.018402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:29.019405 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:29.019418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.020233 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.020240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:29.020450 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:29.021153 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:29.021172 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:29.024254 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":522,"threads_started":1}
W20260927 22:02:29.024794 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:29.014659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:58431 (local address 127.3.159.62:38627)
0927 22:02:29.015127 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:29.015135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:29.015590 (+ 455us) server_negotiation.cc:407] Connection header received
0927 22:02:29.015768 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:29.015774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:29.015874 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:29.016027 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:29.017233 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.018170 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.019587 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.020047 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.020203 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:29.024518 (+ 4315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:58431: BlockingRecv error: recv got EOF from 127.3.159.1:58431 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 27 22:02:29 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:29 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:29.189199 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:29.179064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:47317)
0927 22:02:29.179849 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:29.179869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:29.180076 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:29.180529 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:29.180533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:29.181104 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:29.181277 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:29.181287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.182679 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.182685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:29.183757 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:29.183770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.184665 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.184672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:29.184849 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:29.185677 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:29.185701 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:29.188922 (+ 3221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":422,"threads_started":1}
W20260927 22:02:29.189618 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:29.179258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:47317 (local address 127.3.159.62:38627)
0927 22:02:29.179656 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:29.179666 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:29.179985 (+ 319us) server_negotiation.cc:407] Connection header received
0927 22:02:29.180251 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:29.180257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:29.180340 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:29.180497 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:29.181496 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.182522 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.183927 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.184439 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.184617 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:29.189222 (+ 4605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:47317: BlockingRecv error: recv got EOF from 127.3.159.3:47317 (error 108)
Metrics: {"server-negotiator.queue_time_us":210}
Sep 27 22:02:29 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:29 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:29.643787 13037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:29.630048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:41563)
0927 22:02:29.630721 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:29.630740 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:29.630882 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:29.631329 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:29.631333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:29.631779 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:29.631997 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:29.632008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.633501 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.633506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:29.634552 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:29.634567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.635458 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.635464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:29.635657 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:29.639921 (+ 4264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:29.639947 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:29.643490 (+ 3543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":389,"threads_started":1}
W20260927 22:02:29.644385 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:29.630197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:41563 (local address 127.3.159.62:38627)
0927 22:02:29.630426 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:29.630431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:29.630855 (+ 424us) server_negotiation.cc:407] Connection header received
0927 22:02:29.631056 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:29.631065 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:29.631148 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:29.631271 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:29.632236 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.633119 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.634739 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:29.635228 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:29.635378 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:29.643966 (+ 8588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:41563: BlockingRecv error: recv got EOF from 127.3.159.2:41563 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 27 22:02:30 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:30 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.1@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:30.036808 13038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:30.026463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.1:38553)
0927 22:02:30.027280 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:30.027312 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:30.027506 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:30.027919 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:30.027924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:30.028398 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:30.028638 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:30.028648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.030162 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.030168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:30.031371 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:30.031386 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.032335 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.032340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:30.032474 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:30.033274 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:30.033292 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:30.036529 (+ 3237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":542,"threads_started":1}
W20260927 22:02:30.037612 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:30.026600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:38553 (local address 127.3.159.62:38627)
0927 22:02:30.026870 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:30.026877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:30.027446 (+ 569us) server_negotiation.cc:407] Connection header received
0927 22:02:30.027647 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:30.027652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:30.027735 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:30.027865 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:30.028812 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.029984 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.031601 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.032104 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.032292 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:30.037207 (+ 4915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.1:38553: BlockingRecv error: recv got EOF from 127.3.159.1:38553 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 27 22:02:30 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:30 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.3@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:30.201370 13039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:30.190560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.3:43713)
0927 22:02:30.191609 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:30.191630 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:30.191876 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:30.192444 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:30.192450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:30.192909 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:30.193117 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:30.193127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.194459 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.194466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:30.195450 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:30.195463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.196273 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.196279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:30.196474 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:30.197315 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:30.197339 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:30.200914 (+ 3575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":930,"thread_start_us":696,"threads_started":1}
W20260927 22:02:30.201555 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:30.190902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:43713 (local address 127.3.159.62:38627)
0927 22:02:30.191184 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:30.191192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:30.191820 (+ 628us) server_negotiation.cc:407] Connection header received
0927 22:02:30.192047 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:30.192053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:30.192128 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:30.192253 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:30.193333 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.194295 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.195615 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.196085 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.196226 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:30.201242 (+ 5016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.3:43713: BlockingRecv error: recv got EOF from 127.3.159.3:43713 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 22:02:30 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 22:02:30 dist-test-slave-sg4j krb5kdc[11992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.159.2@KRBTEST.COM for oryx/127.3.159.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 22:02:30.655750 13040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:30.645441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38627 (local address 127.3.159.2:33679)
0927 22:02:30.646158 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:30.646177 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:30.646338 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:30.646801 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:30.646805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:30.647523 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:30.647813 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:30.647824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.649232 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.649241 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:30.650358 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:30.650370 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.651137 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.651144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:30.651331 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:30.652023 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:30.652041 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:30.655269 (+ 3228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.159.62:38627: Server oryx/127.3.159.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":421,"threads_started":1}
W20260927 22:02:30.655896 12863 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:30.645624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:33679 (local address 127.3.159.62:38627)
0927 22:02:30.645824 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:30.645829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:30.646290 (+ 461us) server_negotiation.cc:407] Connection header received
0927 22:02:30.646470 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:30.646475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:30.646546 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:30.646679 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:30.648020 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.649057 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.650508 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.650972 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.651106 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:30.655631 (+ 4525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.159.2:33679: BlockingRecv error: recv got EOF from 127.3.159.2:33679 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 22:02:30.979430 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 12581
I20260927 22:02:30.993994 13042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:30.980908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.3.159.1:39425)
0927 22:02:30.981718 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:30.981727 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:30.981759 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:02:30.981838 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:30.981848 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:30.982238 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:30.982452 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:30.983963 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.985056 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.986472 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:30.987104 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:30.987230 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:30.990171 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:30.990182 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:30.990201 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:30.990245 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:30.993229 (+ 2984us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.159.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:02:30.993680 (+ 451us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54710: Request ticket server kudu/127.3.159.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":451,"threads_started":1}
W20260927 22:02:30.994271 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 12581: Not authorized: Client connection negotiation failed: client connection to 127.3.159.1:39425: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.159.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:02:30.994566 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 12581
I20260927 22:02:31.014412 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 12732
I20260927 22:02:31.029249 13043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:31.015939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.3.159.2:41683)
0927 22:02:31.016734 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:31.016749 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:31.016803 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:02:31.016886 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:31.016912 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:31.017255 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:31.017516 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:31.018998 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:31.020206 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:31.021651 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:31.022306 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:31.022471 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:31.025419 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:31.025435 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:31.025462 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:31.025506 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:31.028441 (+ 2935us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.159.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:02:31.028865 (+ 424us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41416: Request ticket server kudu/127.3.159.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":380,"threads_started":1}
W20260927 22:02:31.029386 13043 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41416: Request ticket server kudu/127.3.159.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:02:31.029470 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 12732: Not authorized: Client connection negotiation failed: client connection to 127.3.159.2:41683: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.159.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:02:31.029740 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 12732
I20260927 22:02:31.047477 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 12862
I20260927 22:02:31.061818 13044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:31.048682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.3.159.3:42995)
0927 22:02:31.049345 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:31.049353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:31.049439 (+ 86us) server_negotiation.cc:407] Connection header received
0927 22:02:31.049542 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:31.049554 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:31.049806 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:31.049995 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:31.051533 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:31.052592 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:31.054027 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:31.054643 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:31.054770 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:31.057852 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:31.057865 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:31.057882 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:31.057943 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:31.060987 (+ 3044us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.159.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 22:02:31.061312 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33184: Request ticket server kudu/127.3.159.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":341,"threads_started":1}
W20260927 22:02:31.061949 13044 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33184: Request ticket server kudu/127.3.159.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 22:02:31.062116 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 12862: Not authorized: Client connection negotiation failed: client connection to 127.3.159.3:42995: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.159.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 22:02:31.062348 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 12862
I20260927 22:02:31.079792 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 12512
I20260927 22:02:31.081617 12574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:31.214684 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 12512
2026-09-27T22:02:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22489 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T22:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:02:31Z Disabled control of system clock
I20260927 22:02:31.301791 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38485
--webserver_interface=127.3.159.62
--webserver_port=0
--builtin_ntp_servers=127.3.159.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:38485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 22:02:31.658322 13049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:31.658659 13049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:31.658803 13049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:31.667829 13049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:02:31.667982 13049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:31.668023 13049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:31.668171 13049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:02:31.668246 13049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:02:31.678802 13049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:38485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:31.680413 13049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:31.682581 13049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:31.696573 13057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:31.696656 13055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:31.696667 13054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:31.697588 13049 server_base.cc:1034] running on GCE node
I20260927 22:02:31.698640 13049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:31.700592 13049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:31.701876 13049 hybrid_clock.cc:648] HybridClock initialized: now 1790546551701822 us; error 58 us; skew 500 ppm
I20260927 22:02:31.702359 13049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:02:31.705289 13049 webserver.cc:460] Webserver started at http://127.3.159.62:42323/ using document root <none> and password file <none>
I20260927 22:02:31.706207 13049 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:31.706347 13049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:31.706735 13049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:31.710538 13049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "b632d8855d004ecc9fc8901cb5323a76"
format_stamp: "Formatted at 2026-09-27 22:02:31 on dist-test-slave-sg4j"
server_key: "91f4e95a10643c1480fbd542551e4bc5"
server_key_iv: "ad8ce29d933b18fbbf1a43db7959b54b"
server_key_version: "encryptionkey@0"
I20260927 22:02:31.711529 13049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "b632d8855d004ecc9fc8901cb5323a76"
format_stamp: "Formatted at 2026-09-27 22:02:31 on dist-test-slave-sg4j"
server_key: "91f4e95a10643c1480fbd542551e4bc5"
server_key_iv: "ad8ce29d933b18fbbf1a43db7959b54b"
server_key_version: "encryptionkey@0"
I20260927 22:02:31.718425 13049 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 22:02:31.723480 13063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:31.725468 13049 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:02:31.725729 13049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "b632d8855d004ecc9fc8901cb5323a76"
format_stamp: "Formatted at 2026-09-27 22:02:31 on dist-test-slave-sg4j"
server_key: "91f4e95a10643c1480fbd542551e4bc5"
server_key_iv: "ad8ce29d933b18fbbf1a43db7959b54b"
server_key_version: "encryptionkey@0"
I20260927 22:02:31.726023 13049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:31.757640 13049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:31.758780 13049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:31.759140 13049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:31.782124 13049 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:38485
I20260927 22:02:31.782153 13114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:38485 every 8 connection(s)
I20260927 22:02:31.783838 13049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:02:31.788237 13115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:31.789196 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 13049
I20260927 22:02:31.789469 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:02:31.790292 3708 external_mini_cluster.cc:1428] Setting key bbdec3703a4e163eaad1ff687f3461ef
I20260927 22:02:31.805116 13115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76: Bootstrap starting.
I20260927 22:02:31.807940 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:31.793108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.3.159.62:38485)
0927 22:02:31.794659 (+ 1551us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:31.794675 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:31.794731 (+ 56us) server_negotiation.cc:407] Connection header received
0927 22:02:31.796602 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:31.796630 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:31.797252 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:31.797727 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:31.798611 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:31.800026 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:31.802074 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:31.802870 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:31.803018 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:31.804609 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:31.804619 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:31.804643 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:31.805650 (+ 1007us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:31.805787 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:31.805813 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:31.805923 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:31.806232 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 22:02:31.806673 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":542,"threads_started":1}
I20260927 22:02:31.809686 13115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:31.811004 13115 log.cc:824] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76: Log is configured to *not* fsync() on all Append() calls
I20260927 22:02:31.814852 13115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76: No bootstrap required, opened a new log
I20260927 22:02:31.823086 13115 raft_consensus.cc:348] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b632d8855d004ecc9fc8901cb5323a76" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38485 } }
I20260927 22:02:31.823523 13115 raft_consensus.cc:374] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:31.823652 13115 raft_consensus.cc:729] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b632d8855d004ecc9fc8901cb5323a76, State: Initialized, Role: FOLLOWER
I20260927 22:02:31.824316 13115 consensus_queue.cc:260] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b632d8855d004ecc9fc8901cb5323a76" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38485 } }
I20260927 22:02:31.824571 13115 raft_consensus.cc:388] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:02:31.824717 13115 raft_consensus.cc:482] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:02:31.824884 13115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:31.827250 13115 raft_consensus.cc:504] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b632d8855d004ecc9fc8901cb5323a76" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38485 } }
I20260927 22:02:31.827842 13115 leader_election.cc:302] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b632d8855d004ecc9fc8901cb5323a76; no voters:
I20260927 22:02:31.828663 13115 leader_election.cc:288] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:02:31.828948 13120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:02:31.830367 13120 raft_consensus.cc:686] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [term 1 LEADER]: Becoming Leader. State: Replica: b632d8855d004ecc9fc8901cb5323a76, State: Running, Role: LEADER
I20260927 22:02:31.831241 13115 sys_catalog.cc:564] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:02:31.831166 13120 consensus_queue.cc:237] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b632d8855d004ecc9fc8901cb5323a76" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38485 } }
I20260927 22:02:31.835512 13121 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b632d8855d004ecc9fc8901cb5323a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b632d8855d004ecc9fc8901cb5323a76" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38485 } } }
I20260927 22:02:31.835561 13122 sys_catalog.cc:455] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b632d8855d004ecc9fc8901cb5323a76. Latest consensus state: current_term: 1 leader_uuid: "b632d8855d004ecc9fc8901cb5323a76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b632d8855d004ecc9fc8901cb5323a76" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38485 } } }
I20260927 22:02:31.836875 13121 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:31.836974 13122 sys_catalog.cc:458] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76 [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:31.838364 13129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:02:31.845767 13129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:02:31.859555 13129 catalog_manager.cc:1347] Generated new cluster ID: 943b54351b784459aae77fcfcea8aaed
I20260927 22:02:31.859747 13129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:02:31.880472 13129 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:02:31.882547 13129 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:02:31.895291 13129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b632d8855d004ecc9fc8901cb5323a76: Generated new TSK 0
I20260927 22:02:31.896554 13129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:02:31.908020 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38485
--builtin_ntp_servers=127.3.159.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 22:02:32.313321 13139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:32.313670 13139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:32.313755 13139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:32.321532 13139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:32.321656 13139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:32.321765 13139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:02:32.331683 13139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:32.333186 13139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:32.335263 13139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:32.354765 13145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:32.355103 13144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:32.354988 13147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:32.356720 13139 server_base.cc:1034] running on GCE node
I20260927 22:02:32.357656 13139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:32.358845 13139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:32.362310 13139 hybrid_clock.cc:648] HybridClock initialized: now 1790546552362197 us; error 70 us; skew 500 ppm
I20260927 22:02:32.363073 13139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:02:32.366775 13139 webserver.cc:460] Webserver started at http://127.3.159.1:38641/ using document root <none> and password file <none>
I20260927 22:02:32.367901 13139 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:32.368021 13139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:32.368460 13139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:32.372237 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "15a6d0c0ea604944a238e827842a4d29"
format_stamp: "Formatted at 2026-09-27 22:02:32 on dist-test-slave-sg4j"
server_key: "ace87e528d15fa2e490465eb9e88598b"
server_key_iv: "38dba1dbd9a18a3f51bd9bb2bfdacbc4"
server_key_version: "encryptionkey@0"
I20260927 22:02:32.373169 13139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "15a6d0c0ea604944a238e827842a4d29"
format_stamp: "Formatted at 2026-09-27 22:02:32 on dist-test-slave-sg4j"
server_key: "ace87e528d15fa2e490465eb9e88598b"
server_key_iv: "38dba1dbd9a18a3f51bd9bb2bfdacbc4"
server_key_version: "encryptionkey@0"
I20260927 22:02:32.379737 13139 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260927 22:02:32.384601 13153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:32.386580 13139 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 22:02:32.386816 13139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "15a6d0c0ea604944a238e827842a4d29"
format_stamp: "Formatted at 2026-09-27 22:02:32 on dist-test-slave-sg4j"
server_key: "ace87e528d15fa2e490465eb9e88598b"
server_key_iv: "38dba1dbd9a18a3f51bd9bb2bfdacbc4"
server_key_version: "encryptionkey@0"
I20260927 22:02:32.387079 13139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:32.407224 13139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:32.408250 13139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:32.408550 13139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:32.409770 13139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:32.411417 13139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:32.411547 13139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:32.411675 13139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:32.411768 13139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:32.450812 13139 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:44557
I20260927 22:02:32.450932 13265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:44557 every 8 connection(s)
I20260927 22:02:32.452527 13139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:02:32.459498 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 13139
I20260927 22:02:32.459785 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:02:32.460935 3708 external_mini_cluster.cc:1428] Setting key 86c25478a73fd004632e4fc1b4a273a1
I20260927 22:02:32.465117 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38485
--builtin_ntp_servers=127.3.159.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:02:32.470127 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:32.456236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:58349 (local address 127.3.159.62:38485)
0927 22:02:32.456879 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:32.456884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:32.458237 (+ 1353us) server_negotiation.cc:407] Connection header received
0927 22:02:32.461097 (+ 2860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:32.461104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:32.461234 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:32.461410 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:32.463487 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:32.464327 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:32.466514 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:32.466986 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:32.467144 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:32.469062 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:32.469070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:32.469074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:32.469099 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:32.469147 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:32.469160 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:32.469264 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:32.469590 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 22:02:32.469713 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":334,"threads_started":1}
I20260927 22:02:32.471531 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:32.456735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38485 (local address 127.3.159.1:58349)
0927 22:02:32.458018 (+ 1283us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:32.458079 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:32.460645 (+ 2566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:32.461752 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:32.461764 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:32.462001 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:32.463234 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:32.463268 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:32.464535 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:32.464540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:32.466263 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:32.466292 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:32.467301 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:32.467310 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:32.467599 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:32.468579 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:02:32.468582 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:32.468639 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:02:32.468754 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:32.469263 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:32.469276 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:32.469290 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 22:02:32.469568 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 22:02:32.470135 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":364,"threads_started":1}
I20260927 22:02:32.473538 13266 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38485
I20260927 22:02:32.473919 13266 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:32.474877 13266 heartbeater.cc:511] Master 127.3.159.62:38485 requested a full tablet report, sending...
I20260927 22:02:32.477502 13080 ts_manager.cc:194] Registered new tserver with Master: 15a6d0c0ea604944a238e827842a4d29 (127.3.159.1:44557)
I20260927 22:02:32.479206 13080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.1:58349
W20260927 22:02:32.840063 13270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:32.840433 13270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:32.840582 13270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:32.848621 13270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:32.848814 13270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:32.848986 13270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:02:32.858844 13270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:32.860275 13270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:32.862254 13270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:32.878319 13275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:32.878465 13278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:32.878437 13276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:32.880788 13270 server_base.cc:1034] running on GCE node
I20260927 22:02:32.881817 13270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:32.883330 13270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:32.884712 13270 hybrid_clock.cc:648] HybridClock initialized: now 1790546552884648 us; error 55 us; skew 500 ppm
I20260927 22:02:32.885156 13270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:02:32.888370 13270 webserver.cc:460] Webserver started at http://127.3.159.2:45981/ using document root <none> and password file <none>
I20260927 22:02:32.889338 13270 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:32.889542 13270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:32.889964 13270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:32.894279 13270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "fe8f9c132afa4803986db2c99bde35da"
format_stamp: "Formatted at 2026-09-27 22:02:32 on dist-test-slave-sg4j"
server_key: "f00a8b836af20e48d70fe999458738ae"
server_key_iv: "8dfbe63a95864940c8cb64496a62bde9"
server_key_version: "encryptionkey@0"
I20260927 22:02:32.895236 13270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "fe8f9c132afa4803986db2c99bde35da"
format_stamp: "Formatted at 2026-09-27 22:02:32 on dist-test-slave-sg4j"
server_key: "f00a8b836af20e48d70fe999458738ae"
server_key_iv: "8dfbe63a95864940c8cb64496a62bde9"
server_key_version: "encryptionkey@0"
I20260927 22:02:32.902227 13270 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260927 22:02:32.910854 13284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:32.912992 13270 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260927 22:02:32.913311 13270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "fe8f9c132afa4803986db2c99bde35da"
format_stamp: "Formatted at 2026-09-27 22:02:32 on dist-test-slave-sg4j"
server_key: "f00a8b836af20e48d70fe999458738ae"
server_key_iv: "8dfbe63a95864940c8cb64496a62bde9"
server_key_version: "encryptionkey@0"
I20260927 22:02:32.913686 13270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:32.943770 13270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:32.944955 13270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:32.945312 13270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:32.946650 13270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:32.948900 13270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:32.949004 13270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:32.949093 13270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:32.949170 13270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:32.991230 13270 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:33107
I20260927 22:02:32.991287 13396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:33107 every 8 connection(s)
I20260927 22:02:32.992990 13270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:02:32.994719 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 13270
I20260927 22:02:32.994961 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:02:32.995493 3708 external_mini_cluster.cc:1428] Setting key da20a1a940d82462fd25c3b36fad1284
I20260927 22:02:32.999603 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38485
--builtin_ntp_servers=127.3.159.20:33527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 22:02:33.015602 13399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:32.996483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:48437 (local address 127.3.159.62:38485)
0927 22:02:32.997877 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:32.997886 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:32.999056 (+ 1170us) server_negotiation.cc:407] Connection header received
0927 22:02:33.002816 (+ 3760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:33.002823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:33.002928 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:33.003077 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:33.005109 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.006393 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.010774 (+ 4381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.011425 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.011602 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:33.014389 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:33.014399 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:33.014408 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:33.014440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:33.014506 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:33.014529 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:33.014647 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:33.014996 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 22:02:33.015127 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1221,"thread_start_us":1083,"threads_started":1}
I20260927 22:02:33.017194 13400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:32.997129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38485 (local address 127.3.159.2:48437)
0927 22:02:32.998812 (+ 1683us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:32.998878 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:33.002241 (+ 3363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:33.003451 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:33.003468 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:33.003651 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:33.004781 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:33.004816 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.006670 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.006711 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:33.010460 (+ 3749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:33.010486 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.011792 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.011805 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:33.012251 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:33.013912 (+ 1661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:02:33.013915 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:33.013972 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:02:33.014091 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:33.014677 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:33.014690 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:33.014700 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 22:02:33.014953 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 22:02:33.015481 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":570,"threads_started":1}
I20260927 22:02:33.019230 13397 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38485
I20260927 22:02:33.019646 13397 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:33.020740 13397 heartbeater.cc:511] Master 127.3.159.62:38485 requested a full tablet report, sending...
I20260927 22:02:33.022751 13080 ts_manager.cc:194] Registered new tserver with Master: fe8f9c132afa4803986db2c99bde35da (127.3.159.2:33107)
I20260927 22:02:33.023789 13080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.2:48437
W20260927 22:02:33.383692 13401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:33.384074 13401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:33.384231 13401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:33.392805 13401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:33.392949 13401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:33.393129 13401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:02:33.403614 13401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:33.405352 13401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:33.407660 13401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:33.423938 13409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:33.424081 13407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:33.424098 13406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:33.425235 13401 server_base.cc:1034] running on GCE node
I20260927 22:02:33.426076 13401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:33.427193 13401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:33.428476 13401 hybrid_clock.cc:648] HybridClock initialized: now 1790546553428406 us; error 40 us; skew 500 ppm
I20260927 22:02:33.429025 13401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:02:33.432987 13401 webserver.cc:460] Webserver started at http://127.3.159.3:37229/ using document root <none> and password file <none>
I20260927 22:02:33.433945 13401 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:33.434053 13401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:33.434373 13401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:33.437631 13401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "5c9910fdcd234b71a74ee03013e2af59"
format_stamp: "Formatted at 2026-09-27 22:02:33 on dist-test-slave-sg4j"
server_key: "3ba429b24b6ae7fb4fd8aebf3c55dc76"
server_key_iv: "04fe0b4ce4c3d4d7c9a20ff1d9a5156c"
server_key_version: "encryptionkey@0"
I20260927 22:02:33.438514 13401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "5c9910fdcd234b71a74ee03013e2af59"
format_stamp: "Formatted at 2026-09-27 22:02:33 on dist-test-slave-sg4j"
server_key: "3ba429b24b6ae7fb4fd8aebf3c55dc76"
server_key_iv: "04fe0b4ce4c3d4d7c9a20ff1d9a5156c"
server_key_version: "encryptionkey@0"
I20260927 22:02:33.445035 13401 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 22:02:33.450140 13415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:33.452169 13401 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260927 22:02:33.452384 13401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "5c9910fdcd234b71a74ee03013e2af59"
format_stamp: "Formatted at 2026-09-27 22:02:33 on dist-test-slave-sg4j"
server_key: "3ba429b24b6ae7fb4fd8aebf3c55dc76"
server_key_iv: "04fe0b4ce4c3d4d7c9a20ff1d9a5156c"
server_key_version: "encryptionkey@0"
I20260927 22:02:33.452664 13401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:33.472009 13401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:33.473133 13401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:33.473532 13401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:33.474736 13401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:33.476456 13401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:33.476552 13401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:33.476722 13401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:33.476802 13401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:33.482811 13266 heartbeater.cc:503] Master 127.3.159.62:38485 was elected leader, sending a full tablet report...
I20260927 22:02:33.514940 13401 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:41953
I20260927 22:02:33.515139 13527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:41953 every 8 connection(s)
I20260927 22:02:33.516613 13401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:02:33.524493 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 13401
I20260927 22:02:33.524715 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:02:33.525190 3708 external_mini_cluster.cc:1428] Setting key 118e03986140cdd165f28495167ff65c
I20260927 22:02:33.532567 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:33.519826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:51469 (local address 127.3.159.62:38485)
0927 22:02:33.520419 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:33.520426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:33.522014 (+ 1588us) server_negotiation.cc:407] Connection header received
0927 22:02:33.524139 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:33.524157 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:33.524295 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:33.524469 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:33.526103 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.527189 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.529502 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.529898 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.530020 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:33.531636 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:33.531641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:33.531645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:33.531663 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:33.531713 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:33.531726 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:33.531879 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:33.532123 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 22:02:33.532233 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":255,"threads_started":1}
I20260927 22:02:33.533875 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:33.520431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38485 (local address 127.3.159.3:51469)
0927 22:02:33.521791 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:33.521839 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:33.523717 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:33.524862 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:33.524873 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:33.524989 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:33.525864 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:33.525891 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.527363 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.527372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:33.529292 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:33.529306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.530114 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.530119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:33.530420 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:33.531239 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:02:33.531242 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:33.531293 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:02:33.531413 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:33.531885 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:33.531894 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:33.531901 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:02:33.532063 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 22:02:33.532490 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":370,"threads_started":1}
I20260927 22:02:33.535601 13528 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38485
I20260927 22:02:33.536037 13528 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:33.537107 13528 heartbeater.cc:511] Master 127.3.159.62:38485 requested a full tablet report, sending...
I20260927 22:02:33.538934 13080 ts_manager.cc:194] Registered new tserver with Master: 5c9910fdcd234b71a74ee03013e2af59 (127.3.159.3:41953)
I20260927 22:02:33.540083 13080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.3:51469
I20260927 22:02:33.541960 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:02:33.549506 13530 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:33.547687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.3.159.62:38485)
0927 22:02:33.547858 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:33.547862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:33.548374 (+ 512us) server_negotiation.cc:407] Connection header received
0927 22:02:33.548596 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:33.548601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:33.548677 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:33.548790 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:33.549206 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52744: BlockingRecv error: recv got EOF from 127.0.0.1:52744 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 22:02:33.550616 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 13139
I20260927 22:02:33.559813 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:33.551712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.3.159.1:44557)
0927 22:02:33.552371 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:33.552383 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:33.552421 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:02:33.552499 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:33.552507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:33.552789 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:33.552954 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:33.553472 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.554549 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.556012 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.556651 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.556810 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:33.558093 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:33.558102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:33.558121 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:33.558873 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:33.558929 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:33.558950 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:33.559102 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:33.559252 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 22:02:33.559414 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":331,"threads_started":1}
I20260927 22:02:33.561502 13261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:33.652331 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 13139
I20260927 22:02:33.670437 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 13270
I20260927 22:02:33.679997 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:33.671573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.3.159.2:33107)
0927 22:02:33.672280 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:33.672289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:33.672326 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:02:33.672403 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:33.672411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:33.672736 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:33.672949 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:33.673474 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.674604 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.675982 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.676691 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.676852 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:33.678212 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:33.678221 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:33.678240 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:33.679063 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:33.679122 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:33.679143 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:33.679320 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:33.679402 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 22:02:33.679572 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":370,"threads_started":1}
I20260927 22:02:33.681700 13392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:33.772073 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 13270
I20260927 22:02:33.791615 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 13401
I20260927 22:02:33.802017 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:33.792921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.3.159.3:41953)
0927 22:02:33.794223 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:33.794232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:33.794278 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:02:33.794362 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:33.794370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:33.794619 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:33.794792 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:33.795405 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.796493 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.798145 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:33.798816 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:33.798953 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:33.800197 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:33.800207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:33.800222 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:33.800913 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:33.800975 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:33.800999 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:33.801114 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:33.801349 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 22:02:33.801639 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":531,"threads_started":1}
I20260927 22:02:33.803932 13523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:33.895264 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 13401
I20260927 22:02:33.914139 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 13049
I20260927 22:02:33.915928 13110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:34.024629 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 13049
2026-09-27T22:02:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2807 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T22:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:02:34Z Disabled control of system clock
I20260927 22:02:34.114477 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:41335
--webserver_interface=127.3.159.62
--webserver_port=0
--builtin_ntp_servers=127.3.159.20:44099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:41335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:02:34.523363 13548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:34.523885 13548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:34.524042 13548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:34.532524 13548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:02:34.532689 13548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:34.532732 13548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:34.532855 13548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:02:34.532948 13548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:02:34.543634 13548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:41335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:41335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:34.545483 13548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:34.547784 13548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:34.563272 13554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:34.563426 13553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:34.563477 13556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:34.564709 13548 server_base.cc:1034] running on GCE node
I20260927 22:02:34.566000 13548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:34.568073 13548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:34.569341 13548 hybrid_clock.cc:648] HybridClock initialized: now 1790546554569258 us; error 78 us; skew 500 ppm
I20260927 22:02:34.569903 13548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:02:34.573913 13548 webserver.cc:460] Webserver started at http://127.3.159.62:33411/ using document root <none> and password file <none>
I20260927 22:02:34.574995 13548 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:34.575157 13548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:34.575611 13548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:34.579612 13548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "12c49ff0de124577bf09c548bc1a9629"
format_stamp: "Formatted at 2026-09-27 22:02:34 on dist-test-slave-sg4j"
server_key: "826baf4ee2673166d145f76df09ed4f2"
server_key_iv: "e950d3f6bf46964f9e0520a543b35fc2"
server_key_version: "encryptionkey@0"
I20260927 22:02:34.580636 13548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "12c49ff0de124577bf09c548bc1a9629"
format_stamp: "Formatted at 2026-09-27 22:02:34 on dist-test-slave-sg4j"
server_key: "826baf4ee2673166d145f76df09ed4f2"
server_key_iv: "e950d3f6bf46964f9e0520a543b35fc2"
server_key_version: "encryptionkey@0"
I20260927 22:02:34.587975 13548 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260927 22:02:34.593578 13562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:34.596086 13548 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:02:34.596405 13548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "12c49ff0de124577bf09c548bc1a9629"
format_stamp: "Formatted at 2026-09-27 22:02:34 on dist-test-slave-sg4j"
server_key: "826baf4ee2673166d145f76df09ed4f2"
server_key_iv: "e950d3f6bf46964f9e0520a543b35fc2"
server_key_version: "encryptionkey@0"
I20260927 22:02:34.596707 13548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:34.604522 13548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:34.605736 13548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:34.606104 13548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:34.629515 13548 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:41335
I20260927 22:02:34.629683 13613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:41335 every 8 connection(s)
I20260927 22:02:34.631233 13548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:02:34.635735 13614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:34.636825 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 13548
I20260927 22:02:34.637056 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:02:34.637951 3708 external_mini_cluster.cc:1428] Setting key a8418564c84d1b4cfb6fdd47dab4fed8
I20260927 22:02:34.649763 13614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629: Bootstrap starting.
I20260927 22:02:34.653178 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:34.640891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.3.159.62:41335)
0927 22:02:34.642388 (+ 1497us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:34.642405 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:34.642469 (+ 64us) server_negotiation.cc:407] Connection header received
0927 22:02:34.644518 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:34.644550 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:34.645164 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:34.645692 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:34.650896 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:34.650905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:34.650925 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:34.650946 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:34.651038 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:34.651061 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:34.651321 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:34.651463 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 22:02:34.651905 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":521,"threads_started":1}
I20260927 22:02:34.654785 13614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:34.656140 13614 log.cc:824] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629: Log is configured to *not* fsync() on all Append() calls
I20260927 22:02:34.659893 13614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629: No bootstrap required, opened a new log
I20260927 22:02:34.668270 13614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c49ff0de124577bf09c548bc1a9629" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41335 } }
I20260927 22:02:34.668766 13614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:34.668886 13614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c49ff0de124577bf09c548bc1a9629, State: Initialized, Role: FOLLOWER
I20260927 22:02:34.669661 13614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c49ff0de124577bf09c548bc1a9629" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41335 } }
I20260927 22:02:34.669970 13614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:02:34.670120 13614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:02:34.670303 13614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:34.673418 13614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c49ff0de124577bf09c548bc1a9629" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41335 } }
I20260927 22:02:34.674029 13614 leader_election.cc:302] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12c49ff0de124577bf09c548bc1a9629; no voters:
I20260927 22:02:34.674772 13614 leader_election.cc:288] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:02:34.675074 13619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:02:34.676056 13619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [term 1 LEADER]: Becoming Leader. State: Replica: 12c49ff0de124577bf09c548bc1a9629, State: Running, Role: LEADER
I20260927 22:02:34.676823 13619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c49ff0de124577bf09c548bc1a9629" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41335 } }
I20260927 22:02:34.677310 13614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:02:34.680850 13621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c49ff0de124577bf09c548bc1a9629. Latest consensus state: current_term: 1 leader_uuid: "12c49ff0de124577bf09c548bc1a9629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c49ff0de124577bf09c548bc1a9629" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41335 } } }
I20260927 22:02:34.681416 13621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:34.681965 13620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12c49ff0de124577bf09c548bc1a9629" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c49ff0de124577bf09c548bc1a9629" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 41335 } } }
I20260927 22:02:34.682507 13620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629 [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:34.684832 13629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:02:34.691735 13629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:02:34.703666 13629 catalog_manager.cc:1347] Generated new cluster ID: 927163ec7ab64dc4b35ba6f3350b0d56
I20260927 22:02:34.703855 13629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:02:34.741365 13629 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:02:34.742072 13629 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:02:34.758024 13629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12c49ff0de124577bf09c548bc1a9629: Generated new TSK 0
I20260927 22:02:34.759347 13629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:02:34.777833 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41335
--builtin_ntp_servers=127.3.159.20:44099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 22:02:35.224124 13638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:35.224540 13638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:35.224650 13638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:35.233800 13638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:35.233960 13638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:35.234102 13638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:02:35.245173 13638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:35.246971 13638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:35.249332 13638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:35.267140 13643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:35.267024 13644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:35.267104 13646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:35.268808 13638 server_base.cc:1034] running on GCE node
I20260927 22:02:35.269841 13638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:35.271204 13638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:35.272500 13638 hybrid_clock.cc:648] HybridClock initialized: now 1790546555272407 us; error 83 us; skew 500 ppm
I20260927 22:02:35.272979 13638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:02:35.276167 13638 webserver.cc:460] Webserver started at http://127.3.159.1:46687/ using document root <none> and password file <none>
I20260927 22:02:35.277117 13638 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:35.277289 13638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:35.277756 13638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:35.281921 13638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "4ec40051bc514fb185267604282aab12"
format_stamp: "Formatted at 2026-09-27 22:02:35 on dist-test-slave-sg4j"
server_key: "489249883738aa5421795b9bece510ff"
server_key_iv: "7cc5b91f5d808efd05a61fa4e5eb92f1"
server_key_version: "encryptionkey@0"
I20260927 22:02:35.283026 13638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "4ec40051bc514fb185267604282aab12"
format_stamp: "Formatted at 2026-09-27 22:02:35 on dist-test-slave-sg4j"
server_key: "489249883738aa5421795b9bece510ff"
server_key_iv: "7cc5b91f5d808efd05a61fa4e5eb92f1"
server_key_version: "encryptionkey@0"
I20260927 22:02:35.290176 13638 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 22:02:35.295042 13652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:35.296895 13638 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:02:35.297142 13638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "4ec40051bc514fb185267604282aab12"
format_stamp: "Formatted at 2026-09-27 22:02:35 on dist-test-slave-sg4j"
server_key: "489249883738aa5421795b9bece510ff"
server_key_iv: "7cc5b91f5d808efd05a61fa4e5eb92f1"
server_key_version: "encryptionkey@0"
I20260927 22:02:35.297452 13638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:35.304746 13638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:35.305852 13638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:35.306164 13638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:35.307314 13638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:35.309060 13638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:35.309178 13638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:35.309321 13638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:35.309432 13638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:35.350857 13638 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:33319
I20260927 22:02:35.350967 13764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:33319 every 8 connection(s)
I20260927 22:02:35.352532 13638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:02:35.360394 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 13638
I20260927 22:02:35.360567 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:02:35.361304 3708 external_mini_cluster.cc:1428] Setting key 62b863a21d12807e0b5371b1c6cf3ad5
I20260927 22:02:35.365592 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41335
--builtin_ntp_servers=127.3.159.20:44099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:02:35.366448 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:35.356444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:57719 (local address 127.3.159.62:41335)
0927 22:02:35.356984 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:35.356989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:35.359531 (+ 2542us) server_negotiation.cc:407] Connection header received
0927 22:02:35.362556 (+ 3025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:35.362564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:35.362660 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:35.362850 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:35.365149 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:35.365160 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:35.365168 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:35.365200 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:35.365281 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:35.365304 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:35.365599 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:35.365892 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 22:02:35.366010 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":311,"threads_started":1}
I20260927 22:02:35.367944 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:35.357558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:41335 (local address 127.3.159.1:57719)
0927 22:02:35.359199 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:35.359276 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:35.362038 (+ 2762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:35.363309 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:35.363379 (+ 70us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:35.363578 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:35.364659 (+ 1081us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:02:35.364662 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:35.364718 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:02:35.364852 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:35.365556 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:35.365568 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:35.365577 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:02:35.366019 (+ 442us) client_negotiation.cc:240] Negotiation successful
0927 22:02:35.366455 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":494,"threads_started":1}
I20260927 22:02:35.370543 13765 heartbeater.cc:347] Connected to a master server at 127.3.159.62:41335
I20260927 22:02:35.371045 13765 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:35.371928 13765 heartbeater.cc:511] Master 127.3.159.62:41335 requested a full tablet report, sending...
I20260927 22:02:35.375178 13579 ts_manager.cc:194] Registered new tserver with Master: 4ec40051bc514fb185267604282aab12 (127.3.159.1:33319)
W20260927 22:02:35.776279 13769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:35.776690 13769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:35.776850 13769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:35.786173 13769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:35.786331 13769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:35.786525 13769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:02:35.798206 13769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:35.799997 13769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:35.802541 13769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:35.819571 13774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:35.819834 13777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:35.819834 13775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:35.820878 13769 server_base.cc:1034] running on GCE node
I20260927 22:02:35.821754 13769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:35.822952 13769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:35.824302 13769 hybrid_clock.cc:648] HybridClock initialized: now 1790546555824232 us; error 50 us; skew 500 ppm
I20260927 22:02:35.824806 13769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:02:35.828143 13769 webserver.cc:460] Webserver started at http://127.3.159.2:36879/ using document root <none> and password file <none>
I20260927 22:02:35.829237 13769 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:35.829449 13769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:35.829871 13769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:35.833555 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "698d21a527624b8187093060fe50f896"
format_stamp: "Formatted at 2026-09-27 22:02:35 on dist-test-slave-sg4j"
server_key: "969df6a3c1b8d2d9b35d8e3039ae5ea3"
server_key_iv: "eccb2ba255b0b2daea996ee6eea77b01"
server_key_version: "encryptionkey@0"
I20260927 22:02:35.834591 13769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "698d21a527624b8187093060fe50f896"
format_stamp: "Formatted at 2026-09-27 22:02:35 on dist-test-slave-sg4j"
server_key: "969df6a3c1b8d2d9b35d8e3039ae5ea3"
server_key_iv: "eccb2ba255b0b2daea996ee6eea77b01"
server_key_version: "encryptionkey@0"
I20260927 22:02:35.841871 13769 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:02:35.846693 13783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:35.848922 13769 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:02:35.849203 13769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "698d21a527624b8187093060fe50f896"
format_stamp: "Formatted at 2026-09-27 22:02:35 on dist-test-slave-sg4j"
server_key: "969df6a3c1b8d2d9b35d8e3039ae5ea3"
server_key_iv: "eccb2ba255b0b2daea996ee6eea77b01"
server_key_version: "encryptionkey@0"
I20260927 22:02:35.849568 13769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:35.857218 13769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:35.858498 13769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:35.858982 13769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:35.860543 13769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:35.863047 13769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:35.863255 13769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:35.863421 13769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:35.863514 13769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:35.906049 13769 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:41451
I20260927 22:02:35.906159 13895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:41451 every 8 connection(s)
I20260927 22:02:35.907651 13769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:02:35.909294 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 13769
I20260927 22:02:35.909538 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:02:35.910189 3708 external_mini_cluster.cc:1428] Setting key bcb7dc89eb92f8f39977a41a13847489
I20260927 22:02:35.914546 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41335
--builtin_ntp_servers=127.3.159.20:44099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 22:02:35.921316 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:35.911624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:50951 (local address 127.3.159.62:41335)
0927 22:02:35.912176 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:35.912183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:35.914246 (+ 2063us) server_negotiation.cc:407] Connection header received
0927 22:02:35.917719 (+ 3473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:35.917726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:35.917802 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:35.918106 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:35.920326 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:35.920332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:35.920336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:35.920360 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:35.920421 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:35.920435 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:35.920578 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:35.920925 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 22:02:35.921010 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":271,"threads_started":1}
I20260927 22:02:35.923063 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:35.912397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:41335 (local address 127.3.159.2:50951)
0927 22:02:35.913962 (+ 1565us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:35.914032 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:35.917126 (+ 3094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:35.918458 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:35.918478 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:35.918672 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:35.919725 (+ 1053us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:02:35.919731 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:35.919806 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:02:35.919968 (+ 162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:35.920556 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:35.920569 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:35.920582 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 22:02:35.920946 (+ 364us) client_negotiation.cc:240] Negotiation successful
0927 22:02:35.921346 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":409,"threads_started":1}
I20260927 22:02:35.926080 13896 heartbeater.cc:347] Connected to a master server at 127.3.159.62:41335
I20260927 22:02:35.926681 13896 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:35.927598 13896 heartbeater.cc:511] Master 127.3.159.62:41335 requested a full tablet report, sending...
I20260927 22:02:35.930079 13579 ts_manager.cc:194] Registered new tserver with Master: 698d21a527624b8187093060fe50f896 (127.3.159.2:41451)
W20260927 22:02:36.342453 13900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:36.342840 13900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:36.342944 13900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:36.359180 13900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:36.359369 13900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:36.359616 13900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:02:36.378101 13900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:41335
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:36.379833 13765 heartbeater.cc:503] Master 127.3.159.62:41335 was elected leader, sending a full tablet report...
I20260927 22:02:36.380322 13900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:36.383242 13900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:36.404580 13908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:36.405202 13905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:36.405216 13906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:36.405987 13900 server_base.cc:1034] running on GCE node
I20260927 22:02:36.406733 13900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:36.407930 13900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:36.409195 13900 hybrid_clock.cc:648] HybridClock initialized: now 1790546556409129 us; error 63 us; skew 500 ppm
I20260927 22:02:36.409667 13900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:02:36.412483 13900 webserver.cc:460] Webserver started at http://127.3.159.3:45265/ using document root <none> and password file <none>
I20260927 22:02:36.413287 13900 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:36.413476 13900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:36.413853 13900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:36.416950 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "dbdb493f2d694182a47998bb260b342c"
format_stamp: "Formatted at 2026-09-27 22:02:36 on dist-test-slave-sg4j"
server_key: "0391290037510ef2ec685707f1eb09f1"
server_key_iv: "04d7fe6d791503659ca6d1e68d05f65c"
server_key_version: "encryptionkey@0"
I20260927 22:02:36.417871 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "dbdb493f2d694182a47998bb260b342c"
format_stamp: "Formatted at 2026-09-27 22:02:36 on dist-test-slave-sg4j"
server_key: "0391290037510ef2ec685707f1eb09f1"
server_key_iv: "04d7fe6d791503659ca6d1e68d05f65c"
server_key_version: "encryptionkey@0"
I20260927 22:02:36.423943 13900 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260927 22:02:36.428310 13914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:36.430408 13900 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 22:02:36.430657 13900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "dbdb493f2d694182a47998bb260b342c"
format_stamp: "Formatted at 2026-09-27 22:02:36 on dist-test-slave-sg4j"
server_key: "0391290037510ef2ec685707f1eb09f1"
server_key_iv: "04d7fe6d791503659ca6d1e68d05f65c"
server_key_version: "encryptionkey@0"
I20260927 22:02:36.430897 13900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:36.438190 13900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:36.439121 13900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:36.439383 13900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:36.440442 13900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:36.442319 13900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:36.442425 13900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:36.442533 13900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:36.442669 13900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:36.482496 13900 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:35129
I20260927 22:02:36.482659 14026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:35129 every 8 connection(s)
I20260927 22:02:36.483949 13900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:02:36.494071 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:36.487348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:34507 (local address 127.3.159.62:41335)
0927 22:02:36.487906 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:36.487911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:36.489247 (+ 1336us) server_negotiation.cc:407] Connection header received
0927 22:02:36.491285 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:36.491292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:36.491374 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:36.491510 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:36.493117 (+ 1607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:36.493123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:36.493128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:36.493152 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:36.493201 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:36.493213 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:36.493349 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:36.493594 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 22:02:36.493710 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":300,"threads_started":1}
I20260927 22:02:36.494065 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 13900
I20260927 22:02:36.494367 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:02:36.494983 3708 external_mini_cluster.cc:1428] Setting key 29bb032a1d7b24d8c6427d2ddbc123db
I20260927 22:02:36.495499 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:36.487768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:41335 (local address 127.3.159.3:34507)
0927 22:02:36.489023 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:36.489073 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:36.490950 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:36.491803 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:36.491813 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:36.491934 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:36.492768 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:02:36.492771 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:36.492819 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:02:36.492922 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:36.493317 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:36.493327 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:36.493336 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 22:02:36.493719 (+ 383us) client_negotiation.cc:240] Negotiation successful
0927 22:02:36.494034 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":424,"threads_started":1}
I20260927 22:02:36.497267 14027 heartbeater.cc:347] Connected to a master server at 127.3.159.62:41335
I20260927 22:02:36.497673 14027 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:36.498407 14027 heartbeater.cc:511] Master 127.3.159.62:41335 requested a full tablet report, sending...
I20260927 22:02:36.500276 13579 ts_manager.cc:194] Registered new tserver with Master: dbdb493f2d694182a47998bb260b342c (127.3.159.3:35129)
I20260927 22:02:36.512075 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 22:02:36.519874 14029 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:02:36.517843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.3.159.62:41335)
0927 22:02:36.518167 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:36.518175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:36.518625 (+ 450us) server_negotiation.cc:407] Connection header received
0927 22:02:36.518792 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:36.518796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:36.518864 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:36.518972 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:36.519554 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41574: BlockingRecv error: recv got EOF from 127.0.0.1:41574 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 22:02:36.520533 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 13638
I20260927 22:02:36.525820 14037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:36.522690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.3.159.1:33319)
0927 22:02:36.523357 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:36.523364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:36.523398 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:02:36.523469 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:36.523476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:36.523693 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:36.523844 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:36.524959 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:36.524968 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:36.524981 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:36.524997 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:36.525054 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:36.525072 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:36.525234 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:36.525351 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 22:02:36.525532 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":320,"threads_started":1}
I20260927 22:02:36.527107 13760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:36.616405 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 13638
I20260927 22:02:36.637835 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 13769
I20260927 22:02:36.642386 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:36.638929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59146 (local address 127.3.159.2:41451)
0927 22:02:36.639678 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:36.639687 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:36.639719 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:02:36.639790 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:36.639797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:36.640045 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:36.640266 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:36.641503 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:36.641512 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:36.641531 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:36.641554 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:36.641604 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:36.641621 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:36.641794 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:36.641938 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 22:02:36.642058 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":326,"threads_started":1}
I20260927 22:02:36.643813 13891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:36.734869 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 13769
I20260927 22:02:36.753520 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 13900
I20260927 22:02:36.758021 14041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:36.754536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45402 (local address 127.3.159.3:35129)
0927 22:02:36.755194 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:36.755201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:36.755236 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:02:36.755309 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:36.755316 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:36.755558 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:36.755755 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:36.756900 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:36.756912 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:36.756931 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:02:36.756953 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:36.757047 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:02:36.757069 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:36.757501 (+ 432us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:36.757615 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 22:02:36.757721 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":346,"threads_started":1}
I20260927 22:02:36.759757 14022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:36.846196 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 13900
I20260927 22:02:36.864912 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 13548
I20260927 22:02:36.866495 13609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:36.974896 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 13548
2026-09-27T22:02:36Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2948 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:02:37 dist-test-slave-sg4j krb5kdc[14046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:02:37 dist-test-slave-sg4j krb5kdc[14046](info): set up 2 sockets
Sep 27 22:02:37 dist-test-slave-sg4j krb5kdc[14046](info): commencing operation
krb5kdc: starting...
W20260927 22:02:38.154875 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:02:38 dist-test-slave-sg4j krb5kdc[14046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:02:38.446794 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:39227
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:39227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:02:38.861889 14059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:38.862305 14059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:38.862479 14059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:38.870822 14059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:02:38.870994 14059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:38.871069 14059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:38.871183 14059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:02:38.871255 14059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:02:38.882781 14059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:39227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:39227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:38.884617 14059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:38.886358 14059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:38.901684 14064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:38.901571 14065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:38.901623 14067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:38.907581 14059 server_base.cc:1034] running on GCE node
I20260927 22:02:38.909150 14059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:38.911526 14059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:38.912801 14059 hybrid_clock.cc:648] HybridClock initialized: now 1790546558912723 us; error 83 us; skew 500 ppm
I20260927 22:02:38.913280 14059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:38 dist-test-slave-sg4j krb5kdc[14046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546558, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:38.917444 14059 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:02:38.919740 14059 webserver.cc:460] Webserver started at http://127.3.159.62:45823/ using document root <none> and password file <none>
I20260927 22:02:38.920686 14059 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:38.920809 14059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:38.921209 14059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:38.925103 14059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "032d10d08fcf4ee5b8fc957987b7756a"
format_stamp: "Formatted at 2026-09-27 22:02:38 on dist-test-slave-sg4j"
server_key: "bc7534fef583cf220995984dc0032f8f"
server_key_iv: "3fae184239fca761d6b5bba589751207"
server_key_version: "encryptionkey@0"
I20260927 22:02:38.926093 14059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "032d10d08fcf4ee5b8fc957987b7756a"
format_stamp: "Formatted at 2026-09-27 22:02:38 on dist-test-slave-sg4j"
server_key: "bc7534fef583cf220995984dc0032f8f"
server_key_iv: "3fae184239fca761d6b5bba589751207"
server_key_version: "encryptionkey@0"
I20260927 22:02:38.933257 14059 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 22:02:38.938056 14074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:38.940094 14059 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260927 22:02:38.940366 14059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "032d10d08fcf4ee5b8fc957987b7756a"
format_stamp: "Formatted at 2026-09-27 22:02:38 on dist-test-slave-sg4j"
server_key: "bc7534fef583cf220995984dc0032f8f"
server_key_iv: "3fae184239fca761d6b5bba589751207"
server_key_version: "encryptionkey@0"
I20260927 22:02:38.940625 14059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:38.969521 14059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:38.978120 14059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:38.978539 14059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:39.004343 14059 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:39227
I20260927 22:02:39.004390 14125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:39227 every 8 connection(s)
I20260927 22:02:39.006392 14059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:02:39.010804 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 14059
I20260927 22:02:39.011081 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:02:39.012120 3708 external_mini_cluster.cc:1428] Setting key 965f1ed4dfa9e50823bfb267ea2905a5
I20260927 22:02:39.012316 14126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:39.030977 14126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a: Bootstrap starting.
Sep 27 22:02:39 dist-test-slave-sg4j krb5kdc[14046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:02:39.035696 14126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:39.037150 14126 log.cc:824] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:02:39.041939 14126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a: No bootstrap required, opened a new log
I20260927 22:02:39.046530 14129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:39.015061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.3.159.62:39227)
0927 22:02:39.016500 (+ 1439us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:39.016523 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:39.016601 (+ 78us) server_negotiation.cc:407] Connection header received
0927 22:02:39.018734 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:39.018776 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:39.019481 (+ 705us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:39.020083 (+ 602us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:39.022520 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:39.024217 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:39.026507 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:39.027882 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:39.028154 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:39.034630 (+ 6476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:39.034644 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:39.034671 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:39.034752 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:39.039746 (+ 4994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:39.040982 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:39.040991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:39.041001 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:39.041303 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:39.041927 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:39.041933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:39.041936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:39.042941 (+ 1005us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:39.043352 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:39.043882 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 22:02:39.044608 (+ 726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":614,"threads_started":1}
I20260927 22:02:39.053171 14126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d10d08fcf4ee5b8fc957987b7756a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39227 } }
I20260927 22:02:39.053804 14126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:39.053928 14126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032d10d08fcf4ee5b8fc957987b7756a, State: Initialized, Role: FOLLOWER
I20260927 22:02:39.054697 14126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d10d08fcf4ee5b8fc957987b7756a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39227 } }
I20260927 22:02:39.054942 14126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:02:39.055356 14126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:02:39.055688 14126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:39.058367 14126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d10d08fcf4ee5b8fc957987b7756a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39227 } }
I20260927 22:02:39.058987 14126 leader_election.cc:302] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032d10d08fcf4ee5b8fc957987b7756a; no voters:
I20260927 22:02:39.059859 14126 leader_election.cc:288] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:02:39.060009 14131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:02:39.061671 14131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [term 1 LEADER]: Becoming Leader. State: Replica: 032d10d08fcf4ee5b8fc957987b7756a, State: Running, Role: LEADER
I20260927 22:02:39.062660 14126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:02:39.062666 14131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d10d08fcf4ee5b8fc957987b7756a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39227 } }
I20260927 22:02:39.066835 14132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "032d10d08fcf4ee5b8fc957987b7756a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d10d08fcf4ee5b8fc957987b7756a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39227 } } }
I20260927 22:02:39.066871 14133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032d10d08fcf4ee5b8fc957987b7756a. Latest consensus state: current_term: 1 leader_uuid: "032d10d08fcf4ee5b8fc957987b7756a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032d10d08fcf4ee5b8fc957987b7756a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 39227 } } }
I20260927 22:02:39.067529 14133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:39.067384 14132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a [sys.catalog]: This master's current role is: LEADER
I20260927 22:02:39.071852 14138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:02:39.079317 14138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:02:39.094957 14138 catalog_manager.cc:1347] Generated new cluster ID: 9d3b97c46a8b45568b08208075be06d6
I20260927 22:02:39.095131 14138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:02:39.107506 14138 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:02:39.109992 14138 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:02:39.120242 14138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a: Generated new TSK 0
I20260927 22:02:39.121843 14138 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:02:39.190292 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:39227
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:02:39.594913 14152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:39.595223 14152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:39.595317 14152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:39.602818 14152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:39.602952 14152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:39.603070 14152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:02:39.613131 14152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:39.614566 14152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:39.615888 14152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:39.630985 14158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:39.631084 14160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:39.631214 14157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:39.632645 14152 server_base.cc:1034] running on GCE node
I20260927 22:02:39.633567 14152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:39.634994 14152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:39.636268 14152 hybrid_clock.cc:648] HybridClock initialized: now 1790546559636195 us; error 72 us; skew 500 ppm
I20260927 22:02:39.636677 14152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:39 dist-test-slave-sg4j krb5kdc[14046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546559, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:39.640733 14152 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:02:39.642608 14152 webserver.cc:460] Webserver started at http://127.3.159.1:42699/ using document root <none> and password file <none>
I20260927 22:02:39.643329 14152 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:39.643440 14152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:39.643733 14152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:39.646986 14152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "a279b3c13de14556942649561386bef2"
format_stamp: "Formatted at 2026-09-27 22:02:39 on dist-test-slave-sg4j"
server_key: "747c672d5531810038366b597152eb0d"
server_key_iv: "934b169401c5a81547968a16470605dd"
server_key_version: "encryptionkey@0"
I20260927 22:02:39.647779 14152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "a279b3c13de14556942649561386bef2"
format_stamp: "Formatted at 2026-09-27 22:02:39 on dist-test-slave-sg4j"
server_key: "747c672d5531810038366b597152eb0d"
server_key_iv: "934b169401c5a81547968a16470605dd"
server_key_version: "encryptionkey@0"
I20260927 22:02:39.653599 14152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:02:39.657563 14167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:39.659294 14152 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:02:39.659521 14152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "a279b3c13de14556942649561386bef2"
format_stamp: "Formatted at 2026-09-27 22:02:39 on dist-test-slave-sg4j"
server_key: "747c672d5531810038366b597152eb0d"
server_key_iv: "934b169401c5a81547968a16470605dd"
server_key_version: "encryptionkey@0"
I20260927 22:02:39.659747 14152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:39.686748 14152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:39.692091 14152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:39.692476 14152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:39.694103 14152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:39.695992 14152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:39.696092 14152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:39.696238 14152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:39.696309 14152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:39.741807 14152 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:44695
I20260927 22:02:39.741874 14279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:44695 every 8 connection(s)
I20260927 22:02:39.743623 14152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:02:39.750970 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 14152
I20260927 22:02:39.751168 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:02:39.751927 3708 external_mini_cluster.cc:1428] Setting key 5e564d077f1bab2a121c41735b78c127
Sep 27 22:02:39 dist-test-slave-sg4j krb5kdc[14046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546559, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:02:39.774366 14282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:39.747236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:39681 (local address 127.3.159.62:39227)
0927 22:02:39.747765 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:39.747770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:39.749138 (+ 1368us) server_negotiation.cc:407] Connection header received
0927 22:02:39.751918 (+ 2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:39.751924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:39.752027 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:39.752180 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:39.754719 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:39.755510 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:39.758743 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:39.759247 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:39.759401 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:39.765642 (+ 6241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:39.765653 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:39.765660 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:39.765709 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:39.769644 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:39.770909 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:39.770918 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:39.770925 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:39.771119 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:39.771919 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:39.771927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:39.771933 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:39.772464 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:39.772648 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:39.773670 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0927 22:02:39.773981 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":293,"threads_started":1}
I20260927 22:02:39.775405 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:39.747874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:39227 (local address 127.3.159.1:39681)
0927 22:02:39.748904 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:39.748960 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:39.751547 (+ 2587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:39.752635 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:39.752653 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:39.753471 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:39.754441 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:39.754469 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:39.755729 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:39.755737 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:39.758417 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:39.758448 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:39.759627 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:39.759637 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:39.760072 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:39.761162 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:39.761210 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:39.765180 (+ 3970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:39.769936 (+ 4756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:39.769948 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:39.769970 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:39.770723 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:39.771360 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:39.771365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:39.771369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:39.771729 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:39.772622 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:39.772630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:39.773083 (+ 453us) client_negotiation.cc:769] Sending connection context
0927 22:02:39.773493 (+ 410us) client_negotiation.cc:240] Negotiation successful
0927 22:02:39.774107 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":400,"threads_started":1}
I20260927 22:02:39.777434 14280 heartbeater.cc:347] Connected to a master server at 127.3.159.62:39227
I20260927 22:02:39.777935 14280 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:39.779090 14280 heartbeater.cc:511] Master 127.3.159.62:39227 requested a full tablet report, sending...
I20260927 22:02:39.782495 14091 ts_manager.cc:194] Registered new tserver with Master: a279b3c13de14556942649561386bef2 (127.3.159.1:44695)
I20260927 22:02:39.784077 14091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:39681
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:02:39.824728 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:39227
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:02:40.257270 14286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:40.257756 14286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:40.257951 14286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:40.267004 14286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:40.267232 14286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:40.267454 14286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:02:40.279166 14286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:40.281075 14286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:40.283421 14286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:40.300983 14291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:40.301087 14292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:40.301061 14294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:40.303313 14286 server_base.cc:1034] running on GCE node
I20260927 22:02:40.304291 14286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:40.305904 14286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:40.307209 14286 hybrid_clock.cc:648] HybridClock initialized: now 1790546560307142 us; error 60 us; skew 500 ppm
I20260927 22:02:40.307658 14286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:40 dist-test-slave-sg4j krb5kdc[14046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:40.312096 14286 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:02:40.314435 14286 webserver.cc:460] Webserver started at http://127.3.159.2:36167/ using document root <none> and password file <none>
I20260927 22:02:40.315428 14286 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:40.315611 14286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:40.316020 14286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:40.319941 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "9fca2617a4014688b2513d7d6b15fc2e"
format_stamp: "Formatted at 2026-09-27 22:02:40 on dist-test-slave-sg4j"
server_key: "0c0e2d79673b3c65d7ef8f3767d4bd3b"
server_key_iv: "8c3f6519eab216dd47ade8ada15d9a81"
server_key_version: "encryptionkey@0"
I20260927 22:02:40.320973 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "9fca2617a4014688b2513d7d6b15fc2e"
format_stamp: "Formatted at 2026-09-27 22:02:40 on dist-test-slave-sg4j"
server_key: "0c0e2d79673b3c65d7ef8f3767d4bd3b"
server_key_iv: "8c3f6519eab216dd47ade8ada15d9a81"
server_key_version: "encryptionkey@0"
I20260927 22:02:40.328291 14286 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260927 22:02:40.333624 14301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:40.335798 14286 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:02:40.336122 14286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "9fca2617a4014688b2513d7d6b15fc2e"
format_stamp: "Formatted at 2026-09-27 22:02:40 on dist-test-slave-sg4j"
server_key: "0c0e2d79673b3c65d7ef8f3767d4bd3b"
server_key_iv: "8c3f6519eab216dd47ade8ada15d9a81"
server_key_version: "encryptionkey@0"
I20260927 22:02:40.336418 14286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:40.368258 14286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:40.375351 14286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:40.375825 14286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:40.377348 14286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:40.379309 14286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:40.379439 14286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:40.379602 14286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:40.379675 14286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:40.423987 14286 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:46139
I20260927 22:02:40.424057 14413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:46139 every 8 connection(s)
I20260927 22:02:40.425752 14286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:02:40.429143 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 14286
I20260927 22:02:40.429432 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:02:40.430076 3708 external_mini_cluster.cc:1428] Setting key 262407534d11164ffdc5a51d4dfe9711
Sep 27 22:02:40 dist-test-slave-sg4j krb5kdc[14046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:02:40.453259 14416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:40.429435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:33403 (local address 127.3.159.62:39227)
0927 22:02:40.430046 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:40.430054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:40.431400 (+ 1346us) server_negotiation.cc:407] Connection header received
0927 22:02:40.433434 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:40.433444 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:40.433554 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:40.433708 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:40.435715 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:40.436944 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:40.438933 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:40.439747 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:40.439940 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:40.446036 (+ 6096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:40.446051 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:40.446056 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:40.446118 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:40.449508 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:40.450585 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:40.450591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:40.450602 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:40.450734 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:40.451210 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:40.451220 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:40.451222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:40.451550 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:40.451718 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:40.452562 (+ 844us) server_negotiation.cc:299] Negotiation successful
0927 22:02:40.452832 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":282,"threads_started":1}
I20260927 22:02:40.454577 14417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:40.430204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:39227 (local address 127.3.159.2:33403)
0927 22:02:40.431163 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:40.431255 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:40.433074 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:40.434047 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:40.434058 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:40.434721 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:40.435462 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:40.435486 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:40.437203 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:40.437208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:40.438722 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:40.438739 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:40.440154 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:40.440163 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:40.440644 (+ 481us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:40.441641 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:40.441672 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:40.445612 (+ 3940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:40.449799 (+ 4187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:40.449810 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:40.449831 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:40.450431 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:40.450850 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:40.450856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:40.450859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:40.451077 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:40.451690 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:40.451701 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:40.452154 (+ 453us) client_negotiation.cc:769] Sending connection context
0927 22:02:40.452467 (+ 313us) client_negotiation.cc:240] Negotiation successful
0927 22:02:40.453139 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":397,"threads_started":1}
I20260927 22:02:40.456561 14414 heartbeater.cc:347] Connected to a master server at 127.3.159.62:39227
I20260927 22:02:40.457105 14414 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
I20260927 22:02:40.458284 14414 heartbeater.cc:511] Master 127.3.159.62:39227 requested a full tablet report, sending...
I20260927 22:02:40.460373 14091 ts_manager.cc:194] Registered new tserver with Master: 9fca2617a4014688b2513d7d6b15fc2e (127.3.159.2:46139)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
I20260927 22:02:40.461606 14091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:33403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:02:40.494457 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:39227
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:02:40.787778 14280 heartbeater.cc:503] Master 127.3.159.62:39227 was elected leader, sending a full tablet report...
W20260927 22:02:40.905812 14420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:02:40.906461 14420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:02:40.906636 14420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:02:40.917171 14420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:02:40.917330 14420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:02:40.917563 14420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:02:40.928893 14420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:39227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:02:40.930711 14420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:02:40.932416 14420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:02:40.948720 14426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:40.949015 14428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:02:40.949158 14425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:02:40.955307 14420 server_base.cc:1034] running on GCE node
I20260927 22:02:40.956182 14420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:02:40.957353 14420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:02:40.958725 14420 hybrid_clock.cc:648] HybridClock initialized: now 1790546560958654 us; error 61 us; skew 500 ppm
I20260927 22:02:40.959173 14420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:02:40 dist-test-slave-sg4j krb5kdc[14046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:02:40.963155 14420 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:02:40.965026 14420 webserver.cc:460] Webserver started at http://127.3.159.3:33231/ using document root <none> and password file <none>
I20260927 22:02:40.965920 14420 fs_manager.cc:359] Metadata directory not provided
I20260927 22:02:40.966068 14420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:02:40.966380 14420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:02:40.969687 14420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "e2f484f9956b43eeb1dc5455681c51cb"
format_stamp: "Formatted at 2026-09-27 22:02:40 on dist-test-slave-sg4j"
server_key: "c7c375587d93c5bd20bd44de246b156a"
server_key_iv: "e98fad77242572ba5b0d70145f8cead0"
server_key_version: "encryptionkey@0"
I20260927 22:02:40.970515 14420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "e2f484f9956b43eeb1dc5455681c51cb"
format_stamp: "Formatted at 2026-09-27 22:02:40 on dist-test-slave-sg4j"
server_key: "c7c375587d93c5bd20bd44de246b156a"
server_key_iv: "e98fad77242572ba5b0d70145f8cead0"
server_key_version: "encryptionkey@0"
I20260927 22:02:40.976549 14420 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 22:02:40.980746 14435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:40.982533 14420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:02:40.982722 14420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "e2f484f9956b43eeb1dc5455681c51cb"
format_stamp: "Formatted at 2026-09-27 22:02:40 on dist-test-slave-sg4j"
server_key: "c7c375587d93c5bd20bd44de246b156a"
server_key_iv: "e98fad77242572ba5b0d70145f8cead0"
server_key_version: "encryptionkey@0"
I20260927 22:02:40.982941 14420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:02:41.003171 14420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:02:41.008460 14420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:02:41.008780 14420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:02:41.009975 14420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:02:41.011543 14420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:02:41.011650 14420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:41.011808 14420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:02:41.011879 14420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:02:41.052891 14420 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:42747
I20260927 22:02:41.052999 14547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:42747 every 8 connection(s)
I20260927 22:02:41.054410 14420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:02:41.055939 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 14420
I20260927 22:02:41.056141 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:02:41.056610 3708 external_mini_cluster.cc:1428] Setting key ede95f7257b9ef970a976ef40e413f40
Sep 27 22:02:41 dist-test-slave-sg4j krb5kdc[14046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:02:41.080219 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.058050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:59483 (local address 127.3.159.62:39227)
0927 22:02:41.058538 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:41.058546 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:41.060164 (+ 1618us) server_negotiation.cc:407] Connection header received
0927 22:02:41.062498 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:41.062503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:41.062610 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:41.062740 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:41.064707 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.065597 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.067330 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.067758 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.067900 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.072250 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:41.072261 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:41.072266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:41.072372 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:41.076232 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:41.077281 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:41.077287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:41.077290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:41.077554 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:41.078116 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:41.078121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:41.078124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:41.078549 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:41.078705 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:41.079587 (+ 882us) server_negotiation.cc:299] Negotiation successful
0927 22:02:41.079862 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":256,"threads_started":1}
I20260927 22:02:41.081187 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.058854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:39227 (local address 127.3.159.3:59483)
0927 22:02:41.059959 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:41.060032 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:41.062169 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:41.063016 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:41.063031 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:41.063762 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:02:41.064405 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.064531 (+ 126us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.065768 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.065772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.067184 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.067198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.068006 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.068013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.068259 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.069014 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:02:41.069048 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:02:41.071839 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:02:41.076490 (+ 4651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:41.076504 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:41.076522 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:41.077099 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:41.077718 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:02:41.077722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:02:41.077725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:02:41.077955 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:02:41.078729 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:02:41.078737 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:02:41.079132 (+ 395us) client_negotiation.cc:769] Sending connection context
0927 22:02:41.079438 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 22:02:41.079886 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":436,"threads_started":1}
I20260927 22:02:41.083276 14548 heartbeater.cc:347] Connected to a master server at 127.3.159.62:39227
I20260927 22:02:41.083822 14548 heartbeater.cc:464] Registering TS with master...
I20260927 22:02:41.085217 14548 heartbeater.cc:511] Master 127.3.159.62:39227 requested a full tablet report, sending...
I20260927 22:02:41.087870 14091 ts_manager.cc:194] Registered new tserver with Master: e2f484f9956b43eeb1dc5455681c51cb (127.3.159.3:42747)
I20260927 22:02:41.088833 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:02:41.089275 14091 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:59483
I20260927 22:02:41.109671 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.094573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.3.159.62:39227)
0927 22:02:41.094962 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:41.094971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:41.095452 (+ 481us) server_negotiation.cc:407] Connection header received
0927 22:02:41.095723 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:41.095729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:41.095837 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:41.095963 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:41.097612 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.098428 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.099785 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.100239 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.100393 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.103203 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:41.103209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:41.103213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:41.103260 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:41.106491 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:41.107317 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:41.107325 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:41.107329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:41.107475 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:41.107939 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:41.107946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:41.107949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:41.108344 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:41.108621 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:41.108986 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 22:02:41.109250 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260927 22:02:41.115104 14091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:02:41.122259 14091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:02:41.154253 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.138313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:44695 (local address 127.0.0.1:54488)
0927 22:02:41.139602 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:41.139672 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:41.139945 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:41.142600 (+ 2655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:41.142612 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:41.142657 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:41.143233 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.143250 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.146739 (+ 3489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.146748 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.150799 (+ 4051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.150821 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.153066 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.153074 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.153406 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.153486 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 22:02:41.153640 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 22:02:41.153857 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"thread_start_us":804,"threads_started":1}
I20260927 22:02:41.154584 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.138390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.3.159.1:44695)
0927 22:02:41.139085 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:41.139096 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:41.140253 (+ 1157us) server_negotiation.cc:407] Connection header received
0927 22:02:41.140594 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:41.140606 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:41.140949 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:41.141182 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:41.143719 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.146482 (+ 2763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.151069 (+ 4587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.152732 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.152875 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.153134 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:41.153770 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 22:02:41.154023 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":344,"threads_started":1}
I20260927 22:02:41.162523 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.140303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:42747 (local address 127.0.0.1:57996)
0927 22:02:41.151487 (+ 11184us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:41.151509 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:41.151671 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:41.152728 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:41.152732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:41.152752 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:41.153353 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.153364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.157468 (+ 4104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.157475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.159830 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.159846 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.161450 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.161458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.161651 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.161693 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:02:41.161790 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 22:02:41.161968 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11009,"mutex_wait_us":28,"thread_start_us":9821,"threads_started":1}
I20260927 22:02:41.161616 14215 tablet_service.cc:1467] Processing CreateTablet for tablet b54ac63d90e1478c8b2ee9d8a14b8eef (DEFAULT_TABLE table=test-table [id=9df3397b635849b8be0bd7df0ba9e9ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:41.163691 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.141032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.3.159.3:42747)
0927 22:02:41.150689 (+ 9657us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:41.150699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:41.151813 (+ 1114us) server_negotiation.cc:407] Connection header received
0927 22:02:41.151930 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:41.151940 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:41.152344 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:41.152588 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:41.153616 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.157231 (+ 3615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.160075 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.161138 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.161315 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.161651 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:41.162914 (+ 1263us) server_negotiation.cc:299] Negotiation successful
0927 22:02:41.163161 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9431,"thread_start_us":9200,"threads_started":1}
I20260927 22:02:41.163733 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.138960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:46139 (local address 127.0.0.1:59736)
0927 22:02:41.150821 (+ 11861us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:41.150866 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:41.151126 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:41.152482 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:41.152487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:41.152512 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:41.153113 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.153131 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.155423 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.155430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.158574 (+ 3144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.158591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.162241 (+ 3650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.162248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.162467 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.162518 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:02:41.163038 (+ 520us) client_negotiation.cc:240] Negotiation successful
0927 22:02:41.163216 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11646,"mutex_wait_us":24,"thread_start_us":12130,"threads_started":1}
I20260927 22:02:41.164084 14215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54ac63d90e1478c8b2ee9d8a14b8eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:41.164644 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.139465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.3.159.2:46139)
0927 22:02:41.150688 (+ 11223us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:41.150699 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:41.151338 (+ 639us) server_negotiation.cc:407] Connection header received
0927 22:02:41.151537 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:41.151549 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:41.151926 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:41.152206 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:41.153356 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.154979 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.158848 (+ 3869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.160592 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.161853 (+ 1261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.162074 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:41.164076 (+ 2002us) server_negotiation.cc:299] Negotiation successful
0927 22:02:41.164327 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11061,"thread_start_us":11243,"threads_started":1}
I20260927 22:02:41.169082 14483 tablet_service.cc:1467] Processing CreateTablet for tablet b54ac63d90e1478c8b2ee9d8a14b8eef (DEFAULT_TABLE table=test-table [id=9df3397b635849b8be0bd7df0ba9e9ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:41.169082 14349 tablet_service.cc:1467] Processing CreateTablet for tablet b54ac63d90e1478c8b2ee9d8a14b8eef (DEFAULT_TABLE table=test-table [id=9df3397b635849b8be0bd7df0ba9e9ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:02:41.171291 14483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54ac63d90e1478c8b2ee9d8a14b8eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:41.171352 14349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b54ac63d90e1478c8b2ee9d8a14b8eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:02:41.178928 14567 tablet_bootstrap.cc:492] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: Bootstrap starting.
I20260927 22:02:41.184167 14567 tablet_bootstrap.cc:654] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:41.185789 14567 log.cc:824] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:02:41.188951 14567 tablet_bootstrap.cc:492] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: No bootstrap required, opened a new log
I20260927 22:02:41.189595 14567 ts_tablet_manager.cc:1397] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 22:02:41.196270 14570 tablet_bootstrap.cc:492] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: Bootstrap starting.
I20260927 22:02:41.197856 14567 raft_consensus.cc:348] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a279b3c13de14556942649561386bef2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44695 } } peers { permanent_uuid: "e2f484f9956b43eeb1dc5455681c51cb" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42747 } } peers { permanent_uuid: "9fca2617a4014688b2513d7d6b15fc2e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 46139 } }
I20260927 22:02:41.198361 14567 raft_consensus.cc:374] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:41.198470 14567 raft_consensus.cc:729] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a279b3c13de14556942649561386bef2, State: Initialized, Role: FOLLOWER
I20260927 22:02:41.199267 14567 consensus_queue.cc:260] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a279b3c13de14556942649561386bef2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44695 } } peers { permanent_uuid: "e2f484f9956b43eeb1dc5455681c51cb" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42747 } } peers { permanent_uuid: "9fca2617a4014688b2513d7d6b15fc2e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 46139 } }
I20260927 22:02:41.199841 14569 tablet_bootstrap.cc:492] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: Bootstrap starting.
I20260927 22:02:41.201860 14570 tablet_bootstrap.cc:654] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:41.203079 14567 ts_tablet_manager.cc:1428] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260927 22:02:41.203416 14570 log.cc:824] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:02:41.204370 14569 tablet_bootstrap.cc:654] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: Neither blocks nor log segments found. Creating new log.
I20260927 22:02:41.205827 14569 log.cc:824] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:02:41.207171 14570 tablet_bootstrap.cc:492] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: No bootstrap required, opened a new log
I20260927 22:02:41.207521 14570 ts_tablet_manager.cc:1397] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 22:02:41.209510 14569 tablet_bootstrap.cc:492] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: No bootstrap required, opened a new log
I20260927 22:02:41.209805 14569 ts_tablet_manager.cc:1397] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 22:02:41.216719 14570 raft_consensus.cc:348] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a279b3c13de14556942649561386bef2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44695 } } peers { permanent_uuid: "e2f484f9956b43eeb1dc5455681c51cb" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42747 } } peers { permanent_uuid: "9fca2617a4014688b2513d7d6b15fc2e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 46139 } }
I20260927 22:02:41.217242 14570 raft_consensus.cc:374] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:41.217365 14570 raft_consensus.cc:729] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fca2617a4014688b2513d7d6b15fc2e, State: Initialized, Role: FOLLOWER
I20260927 22:02:41.217475 14569 raft_consensus.cc:348] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a279b3c13de14556942649561386bef2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44695 } } peers { permanent_uuid: "e2f484f9956b43eeb1dc5455681c51cb" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42747 } } peers { permanent_uuid: "9fca2617a4014688b2513d7d6b15fc2e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 46139 } }
I20260927 22:02:41.217948 14569 raft_consensus.cc:374] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:02:41.218055 14569 raft_consensus.cc:729] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f484f9956b43eeb1dc5455681c51cb, State: Initialized, Role: FOLLOWER
I20260927 22:02:41.218263 14570 consensus_queue.cc:260] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a279b3c13de14556942649561386bef2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44695 } } peers { permanent_uuid: "e2f484f9956b43eeb1dc5455681c51cb" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42747 } } peers { permanent_uuid: "9fca2617a4014688b2513d7d6b15fc2e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 46139 } }
I20260927 22:02:41.218729 14569 consensus_queue.cc:260] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a279b3c13de14556942649561386bef2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44695 } } peers { permanent_uuid: "e2f484f9956b43eeb1dc5455681c51cb" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42747 } } peers { permanent_uuid: "9fca2617a4014688b2513d7d6b15fc2e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 46139 } }
I20260927 22:02:41.220301 14414 heartbeater.cc:503] Master 127.3.159.62:39227 was elected leader, sending a full tablet report...
I20260927 22:02:41.220798 14570 ts_tablet_manager.cc:1428] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: Time spent starting tablet: real 0.013s user 0.007s sys 0.008s
I20260927 22:02:41.222105 14548 heartbeater.cc:503] Master 127.3.159.62:39227 was elected leader, sending a full tablet report...
I20260927 22:02:41.222820 14569 ts_tablet_manager.cc:1428] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260927 22:02:41.247011 14281 tablet.cc:2369] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:02:41.306398 14549 tablet.cc:2369] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:02:41.425333 14571 raft_consensus.cc:482] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:02:41.425799 14571 raft_consensus.cc:504] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a279b3c13de14556942649561386bef2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44695 } } peers { permanent_uuid: "e2f484f9956b43eeb1dc5455681c51cb" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42747 } } peers { permanent_uuid: "9fca2617a4014688b2513d7d6b15fc2e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 46139 } }
I20260927 22:02:41.427997 14571 leader_election.cc:288] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f484f9956b43eeb1dc5455681c51cb (127.3.159.3:42747), 9fca2617a4014688b2513d7d6b15fc2e (127.3.159.2:46139)
W20260927 22:02:41.428644 14415 tablet.cc:2369] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:02:41.438809 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.428772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:47585 (local address 127.3.159.2:46139)
0927 22:02:41.429092 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:41.429103 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:41.430964 (+ 1861us) server_negotiation.cc:407] Connection header received
0927 22:02:41.431431 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:41.431436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:41.431506 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:41.431628 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:41.432541 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.433725 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.436267 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.437616 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.437796 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.437919 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:41.438327 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 22:02:41.438524 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:02:41.439260 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.429450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:43727 (local address 127.3.159.3:42747)
0927 22:02:41.429935 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:41.429945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:41.431130 (+ 1185us) server_negotiation.cc:407] Connection header received
0927 22:02:41.431319 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:41.431326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:41.431409 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:41.431562 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:02:41.432343 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.433684 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.436712 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.438054 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.438226 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.438382 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:41.438693 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 22:02:41.438843 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260927 22:02:41.439312 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.428652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:42747 (local address 127.3.159.1:43727)
0927 22:02:41.430991 (+ 2339us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:41.431008 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:41.431152 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:41.431599 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:41.431608 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:41.431636 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:41.432129 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.432142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.433859 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.433865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.436495 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.436514 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.438321 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.438328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.438518 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.438544 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:02:41.438662 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 22:02:41.438844 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2209,"mutex_wait_us":36,"thread_start_us":1996,"threads_started":1}
I20260927 22:02:41.440385 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.428383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:46139 (local address 127.3.159.1:47585)
0927 22:02:41.430778 (+ 2395us) negotiation.cc:107] Waiting for socket to connect
0927 22:02:41.430809 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 22:02:41.431210 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:02:41.431814 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:02:41.431821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:02:41.431865 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:02:41.432372 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.432385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.433925 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.433930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.436053 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:02:41.436071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.437887 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.437897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:02:41.438097 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.438139 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:02:41.439246 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0927 22:02:41.439943 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2217,"mutex_wait_us":296,"spinlock_wait_cycles":35072,"thread_start_us":1853,"threads_started":1}
I20260927 22:02:41.440987 14503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54ac63d90e1478c8b2ee9d8a14b8eef" candidate_uuid: "a279b3c13de14556942649561386bef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f484f9956b43eeb1dc5455681c51cb" is_pre_election: true
I20260927 22:02:41.441798 14503 raft_consensus.cc:2454] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a279b3c13de14556942649561386bef2 in term 0.
I20260927 22:02:41.441789 14369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54ac63d90e1478c8b2ee9d8a14b8eef" candidate_uuid: "a279b3c13de14556942649561386bef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fca2617a4014688b2513d7d6b15fc2e" is_pre_election: true
I20260927 22:02:41.442323 14369 raft_consensus.cc:2454] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a279b3c13de14556942649561386bef2 in term 0.
I20260927 22:02:41.442793 14169 leader_election.cc:302] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a279b3c13de14556942649561386bef2, e2f484f9956b43eeb1dc5455681c51cb; no voters:
I20260927 22:02:41.443392 14571 raft_consensus.cc:2781] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:02:41.443574 14571 raft_consensus.cc:482] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:02:41.443722 14571 raft_consensus.cc:3037] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:41.446832 14571 raft_consensus.cc:504] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a279b3c13de14556942649561386bef2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44695 } } peers { permanent_uuid: "e2f484f9956b43eeb1dc5455681c51cb" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42747 } } peers { permanent_uuid: "9fca2617a4014688b2513d7d6b15fc2e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 46139 } }
I20260927 22:02:41.447852 14571 leader_election.cc:288] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [CANDIDATE]: Term 1 election: Requested vote from peers e2f484f9956b43eeb1dc5455681c51cb (127.3.159.3:42747), 9fca2617a4014688b2513d7d6b15fc2e (127.3.159.2:46139)
I20260927 22:02:41.448563 14503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54ac63d90e1478c8b2ee9d8a14b8eef" candidate_uuid: "a279b3c13de14556942649561386bef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f484f9956b43eeb1dc5455681c51cb"
I20260927 22:02:41.448644 14369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b54ac63d90e1478c8b2ee9d8a14b8eef" candidate_uuid: "a279b3c13de14556942649561386bef2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fca2617a4014688b2513d7d6b15fc2e"
I20260927 22:02:41.448887 14503 raft_consensus.cc:3037] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:41.448922 14369 raft_consensus.cc:3037] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:02:41.451375 14503 raft_consensus.cc:2454] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a279b3c13de14556942649561386bef2 in term 1.
I20260927 22:02:41.451810 14369 raft_consensus.cc:2454] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a279b3c13de14556942649561386bef2 in term 1.
I20260927 22:02:41.451967 14169 leader_election.cc:302] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a279b3c13de14556942649561386bef2, e2f484f9956b43eeb1dc5455681c51cb; no voters:
I20260927 22:02:41.452401 14571 raft_consensus.cc:2781] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:02:41.453022 14571 raft_consensus.cc:686] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 1 LEADER]: Becoming Leader. State: Replica: a279b3c13de14556942649561386bef2, State: Running, Role: LEADER
I20260927 22:02:41.453635 14571 consensus_queue.cc:237] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a279b3c13de14556942649561386bef2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44695 } } peers { permanent_uuid: "e2f484f9956b43eeb1dc5455681c51cb" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42747 } } peers { permanent_uuid: "9fca2617a4014688b2513d7d6b15fc2e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 46139 } }
I20260927 22:02:41.459873 14091 catalog_manager.cc:5495] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a279b3c13de14556942649561386bef2 (127.3.159.1). New cstate: current_term: 1 leader_uuid: "a279b3c13de14556942649561386bef2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a279b3c13de14556942649561386bef2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 44695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f484f9956b43eeb1dc5455681c51cb" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fca2617a4014688b2513d7d6b15fc2e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 46139 } health_report { overall_health: UNKNOWN } } }
I20260927 22:02:41.534394 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.528467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.3.159.1:44695)
0927 22:02:41.528745 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:41.528750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:41.528773 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:02:41.528967 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:41.528972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:41.529055 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:41.529194 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:02:41.529659 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.530411 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.531934 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.532399 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.532534 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.533098 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:02:41.533746 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:02:41.533867 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:41.534008 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 22:02:41.534081 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260927 22:02:41.544875 14503 raft_consensus.cc:1261] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb [term 1 FOLLOWER]: Refusing update from remote peer a279b3c13de14556942649561386bef2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:41.544955 14369 raft_consensus.cc:1261] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e [term 1 FOLLOWER]: Refusing update from remote peer a279b3c13de14556942649561386bef2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:02:41.546516 14571 consensus_queue.cc:1035] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fca2617a4014688b2513d7d6b15fc2e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 46139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:41.546885 14578 consensus_queue.cc:1035] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f484f9956b43eeb1dc5455681c51cb" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:02:41.562402 14582 mvcc.cc:205] Tried to move back new op lower bound from 7334078716074258432 to 7334078715718352896. Current Snapshot: MvccSnapshot[applied={T|T < 7334078716074258432}]
I20260927 22:02:41.579932 14090 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:02:41.580315 14090 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:02:41.585865 14090 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 032d10d08fcf4ee5b8fc957987b7756a: Sending DeleteTablet for 3 replicas of tablet b54ac63d90e1478c8b2ee9d8a14b8eef
I20260927 22:02:41.587721 14349 tablet_service.cc:1514] Processing DeleteTablet for tablet b54ac63d90e1478c8b2ee9d8a14b8eef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:41 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:59736
I20260927 22:02:41.588016 14483 tablet_service.cc:1514] Processing DeleteTablet for tablet b54ac63d90e1478c8b2ee9d8a14b8eef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:41 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:57996
I20260927 22:02:41.588789 14215 tablet_service.cc:1514] Processing DeleteTablet for tablet b54ac63d90e1478c8b2ee9d8a14b8eef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:02:41 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:54488
I20260927 22:02:41.589490 14591 tablet_replica.cc:330] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: stopping tablet replica
I20260927 22:02:41.589515 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 14152
I20260927 22:02:41.590121 14591 raft_consensus.cc:2229] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:02:41.590797 14591 raft_consensus.cc:2258] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:41.592584 14593 tablet_replica.cc:330] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: stopping tablet replica
I20260927 22:02:41.593197 14593 raft_consensus.cc:2229] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:02:41.594173 14593 raft_consensus.cc:2258] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:41.596875 14591 ts_tablet_manager.cc:1905] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:41.597252 14592 tablet_replica.cc:330] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: stopping tablet replica
I20260927 22:02:41.597886 14592 raft_consensus.cc:2229] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:02:41.598511 14592 raft_consensus.cc:2258] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:02:41.601476 14592 ts_tablet_manager.cc:1905] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:41.604895 14591 ts_tablet_manager.cc:1918] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:02:41.605113 14591 log.cc:1196] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/b54ac63d90e1478c8b2ee9d8a14b8eef
I20260927 22:02:41.605755 14591 ts_tablet_manager.cc:1939] T b54ac63d90e1478c8b2ee9d8a14b8eef P 9fca2617a4014688b2513d7d6b15fc2e: Deleting consensus metadata
I20260927 22:02:41.607347 14593 ts_tablet_manager.cc:1905] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:02:41.607880 14077 catalog_manager.cc:4841] TS 9fca2617a4014688b2513d7d6b15fc2e (127.3.159.2:46139): tablet b54ac63d90e1478c8b2ee9d8a14b8eef (table test-table [id=9df3397b635849b8be0bd7df0ba9e9ec]) successfully deleted
Sep 27 22:02:41 dist-test-slave-sg4j krb5kdc[14046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:02:41.611495 14592 ts_tablet_manager.cc:1918] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:02:41.611721 14592 log.cc:1196] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/b54ac63d90e1478c8b2ee9d8a14b8eef
I20260927 22:02:41.612280 14592 ts_tablet_manager.cc:1939] T b54ac63d90e1478c8b2ee9d8a14b8eef P e2f484f9956b43eeb1dc5455681c51cb: Deleting consensus metadata
I20260927 22:02:41.615613 14593 ts_tablet_manager.cc:1918] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:02:41.615839 14593 log.cc:1196] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/b54ac63d90e1478c8b2ee9d8a14b8eef
I20260927 22:02:41.615868 14076 catalog_manager.cc:4841] TS e2f484f9956b43eeb1dc5455681c51cb (127.3.159.3:42747): tablet b54ac63d90e1478c8b2ee9d8a14b8eef (table test-table [id=9df3397b635849b8be0bd7df0ba9e9ec]) successfully deleted
I20260927 22:02:41.616580 14593 ts_tablet_manager.cc:1939] T b54ac63d90e1478c8b2ee9d8a14b8eef P a279b3c13de14556942649561386bef2: Deleting consensus metadata
I20260927 22:02:41.618750 14076 catalog_manager.cc:4841] TS a279b3c13de14556942649561386bef2 (127.3.159.1:44695): tablet b54ac63d90e1478c8b2ee9d8a14b8eef (table test-table [id=9df3397b635849b8be0bd7df0ba9e9ec]) successfully deleted
I20260927 22:02:41.621301 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.592005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.3.159.1:44695)
0927 22:02:41.592300 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:41.592311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:41.596350 (+ 4039us) server_negotiation.cc:407] Connection header received
0927 22:02:41.597626 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:41.597633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:41.597769 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:41.597862 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:41.600199 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.601525 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.603560 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.604327 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.604440 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.611262 (+ 6822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:41.611290 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:41.611316 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:41.611372 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:41.617628 (+ 6256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:41.618728 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:41.618742 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:41.618746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:41.618975 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:41.619416 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:41.619421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:41.619425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:41.619904 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:41.620282 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:41.620572 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 22:02:41.620851 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 22:02:41.622200 14275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:41.729815 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 14152
I20260927 22:02:41.749584 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 14286
Sep 27 22:02:41 dist-test-slave-sg4j krb5kdc[14046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:02:41.767340 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.750483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.3.159.2:46139)
0927 22:02:41.750712 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:41.750716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:41.750736 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:02:41.750815 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:41.750819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:41.750913 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:41.751019 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:41.752281 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.753089 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.754468 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.754962 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.755079 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.760286 (+ 5207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:41.760311 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:41.760336 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:41.760390 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:41.763873 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:41.764806 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:41.764815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:41.764818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:41.764989 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:41.765653 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:41.765657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:41.765659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:41.766094 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:41.766369 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:41.766648 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 22:02:41.766854 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 22:02:41.768301 14409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:41.867730 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 14286
I20260927 22:02:41.889098 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 14420
Sep 27 22:02:41 dist-test-slave-sg4j krb5kdc[14046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:02:41.909194 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:02:41.890056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.3.159.3:42747)
0927 22:02:41.890333 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 22:02:41.890339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:02:41.890365 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:02:41.890442 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:02:41.890447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:02:41.890528 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:02:41.890635 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:02:41.892389 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.893241 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.895310 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:02:41.895791 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:02:41.895935 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:02:41.901259 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:02:41.901284 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:02:41.901307 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:02:41.901367 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:02:41.905019 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:41.906253 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:41.906263 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:41.906266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:41.906620 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:02:41.907196 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:02:41.907201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:02:41.907203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:02:41.907746 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:02:41.908037 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:02:41.908534 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 22:02:41.908746 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260927 22:02:41.910109 14543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:42.003073 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 14420
I20260927 22:02:42.022049 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 14059
I20260927 22:02:42.023598 14121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:02:42.139289 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 14059
2026-09-27T22:02:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5168 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:02:42 dist-test-slave-sg4j krb5kdc[14601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:02:42 dist-test-slave-sg4j krb5kdc[14601](info): set up 2 sockets
Sep 27 22:02:42 dist-test-slave-sg4j krb5kdc[14601](info): commencing operation
krb5kdc: starting...
W20260927 22:02:43.294868 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:02:43 dist-test-slave-sg4j krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:02:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.159.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.159.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/rangeradmin_127.3.159.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/rangeradmin_127.3.159.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.159.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.159.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/rangerlookup_127.3.159.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/rangerlookup_127.3.159.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.159.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.159.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/HTTP_127.3.159.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/HTTP_127.3.159.20@KRBTEST.COM.keytab.
I20260927 22:02:43.704792 3708 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIPxQZD/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/postgres -l logfile start
2026-09-27 22:02:46.605 UTC [14631] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:02:46.605 UTC [14631] LOG: listening on IPv4 address "127.3.159.20", port 48795
2026-09-27 22:02:46.610 UTC [14631] LOG: listening on Unix socket "/tmp/.s.PGSQL.48795"
2026-09-27 22:02:46.635 UTC [14634] LOG: database system was shut down at 2026-09-27 22:02:44 UTC
2026-09-27 22:02:46.640 UTC [14631] LOG: database system is ready to accept connections
I20260927 22:02:47.646449 3708 mini_postgres.cc:96] Postgres bound to 48795
2026-09-27 22:02:47.667 UTC [14642] FATAL: database "slave" does not exist
127.3.159.20:48795 - accepting connections
I20260927 22:02:47.668905 3708 mini_ranger.cc:161] Starting Ranger...
I20260927 22:02:47.701339 3708 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:02:48.287303 3708 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:02:48.287576 3708 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-admin
2026-09-27 22:02:48,611 [I] DB FLAVOR :POSTGRES
2026-09-27 22:02:48,612 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:02:48,612 [I] Checking connection..
2026-09-27 22:02:48,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:02:48,931 [I] Checking connection passed.
2026-09-27 22:02:48,931 [I] --------- Verifying version history table ---------
2026-09-27 22:02:48,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:02:49,218 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:02:49,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:02:49,516 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:02:49,516 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:02:49,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:02:49.813 UTC [14706] WARNING: there is no transaction in progress
2026-09-27 22:02:49,821 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:02:49,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:02:50,112 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:02:50,113 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:02:50,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:02:50,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:02:50,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-sg4j.c.gcp-upstream.internal','N') ;"
2026-09-27 22:02:51,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:02:51,304 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:02:51,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:02:51,623 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:02:51,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:02:51,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:02:52,186 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:02:52,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:02:53.402 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.424 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.447 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.465 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.485 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.509 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.583 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.591 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.609 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.622 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.635 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.654 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.668 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.678 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.688 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:53.697 UTC [14814] WARNING: there is no transaction in progress
2026-09-27 22:02:54,569 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:02:54,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:02:54,863 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:02:54,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:02:55,165 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:02:55,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:02:55,463 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:02:55,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:02:55,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:02:56,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-sg4j.c.gcp-upstream.internal';"
2026-09-27 22:02:56,333 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:02:56,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:02:56,629 [I] DB_PATCHES have already been applied
I20260927 22:02:56.636529 3708 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIPxQZD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:02:56.636749 3708 mini_ranger.cc:193] Using host: 127.3.159.20
I20260927 22:02:56.656466 3708 mini_ranger.cc:227] Ranger admin URL: http://127.3.159.20:55073
Sep 27, 2026 10:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:02:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:02:57.909636 3708 mini_ranger.cc:160] Time spent starting Ranger: real 10.241s user 0.003s sys 0.056s
Sep 27, 2026 10:03:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:03:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 10:03:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:03:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:03:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:03:23.484452 3708 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.159.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.159.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/rangerkms_127.3.159.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/rangerkms_127.3.159.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.159.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.159.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.159.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/HTTP_127.3.159.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.159.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/HTTP_127.3.159.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:03:23 dist-test-slave-sg4j krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546603, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:03:23.669581 3708 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:03:23.696112 3708 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:03:24.738648 3708 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:03:24.738790 3708 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-kms
I20260927 22:03:24.739002 3708 mini_ranger_kms.cc:227] Using postgres at 127.3.159.20:48795
2026-09-27 22:03:25,267 [I] DB FLAVOR :POSTGRES
2026-09-27 22:03:25,267 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:03:25,267 [I] Checking connection
2026-09-27 22:03:25,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:03:25,546 [I] connection success
2026-09-27 22:03:25,546 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:03:25,546 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:03:25,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:03:25,822 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:03:25,822 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:03:25,823 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:03:25,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:48795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:03:26,121 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:03:26.696760 3708 mini_ranger_kms.cc:310] Created kms service
I20260927 22:03:26.962435 3708 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:03:27.019096 3708 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:03:27 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546603, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:03:27.521376 3708 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:03:27.521854 3708 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:03:27.521932 3708 mini_ranger_kms.cc:240] Using host: 127.3.159.20
I20260927 22:03:27.548799 3708 mini_ranger_kms.cc:276] Ranger KMS PID: 14995
I20260927 22:03:27.549095 3708 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.159.20:34629
Sep 27, 2026 10:03:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:03:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:03:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:03:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:03:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:03:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 22:03:29.922350 3708 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.253s user 0.145s sys 0.413s
I20260927 22:03:29.922693 3708 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:03:29.922961 3708 mini_ranger_kms.cc:401] 127.3.159.20:34629/kms/v1/keys
Sep 27 22:03:30 dist-test-slave-sg4j krb5kdc[14601](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790546610, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.159.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 22:03:30 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546610, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.159.20@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
Sep 27 22:03:31 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546610, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.159.20@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
Sep 27 22:03:33 dist-test-slave-sg4j krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:03:33.641798 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37927
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:37927
--ranger_config_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:34629/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:03:34.053680 15045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:03:34.054021 15045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:03:34.054185 15045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:03:34.063463 15045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:03:34.063709 15045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:03:34.063832 15045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:03:34.063957 15045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:03:34.064214 15045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:03:34.077099 15045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.159.20:34629/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:37927
--ranger_config_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:03:34.078675 15045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:03:34.080109 15045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:03:34.096463 15050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:03:34.096691 15053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:03:34.096616 15051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:03:34.097890 15045 server_base.cc:1034] running on GCE node
I20260927 22:03:34.099006 15045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:03:34.101050 15045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:03:34.102627 15045 hybrid_clock.cc:648] HybridClock initialized: now 1790546614102539 us; error 97 us; skew 500 ppm
I20260927 22:03:34.103204 15045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:03:34 dist-test-slave-sg4j krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:03:34.116103 15045 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:03:34.118592 15045 webserver.cc:460] Webserver started at http://127.3.159.62:38209/ using document root <none> and password file <none>
I20260927 22:03:34.120373 15045 fs_manager.cc:359] Metadata directory not provided
I20260927 22:03:34.120587 15045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:03:34.121025 15045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:03:34 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:03:34.190773 15045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "a08523bda01e472a9232da5b4e586fc2"
format_stamp: "Formatted at 2026-09-27 22:03:34 on dist-test-slave-sg4j"
server_key: "ab1de24b1e8c11807df85fde565f7bff"
server_key_iv: "f0011f5bddb3b4e3a391531b338a0039"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:34.192219 15045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "a08523bda01e472a9232da5b4e586fc2"
format_stamp: "Formatted at 2026-09-27 22:03:34 on dist-test-slave-sg4j"
server_key: "ab1de24b1e8c11807df85fde565f7bff"
server_key_iv: "f0011f5bddb3b4e3a391531b338a0039"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:34.201960 15045 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20260927 22:03:34.225597 15060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:03:34.227952 15045 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:03:34.228229 15045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "a08523bda01e472a9232da5b4e586fc2"
format_stamp: "Formatted at 2026-09-27 22:03:34 on dist-test-slave-sg4j"
server_key: "ab1de24b1e8c11807df85fde565f7bff"
server_key_iv: "f0011f5bddb3b4e3a391531b338a0039"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:34.228511 15045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:03:34.261240 15045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:03:34.266598 15045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:03:34.266978 15045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:03:34.292887 15045 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:37927
I20260927 22:03:34.292943 15111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:37927 every 8 connection(s)
I20260927 22:03:34.294478 15045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:03:34.299567 15112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:03:34.300180 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 15045
I20260927 22:03:34.300469 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:03:34.301142 3708 external_mini_cluster.cc:1428] Setting key 8137c86134a63baa57d275f47c7551d5
I20260927 22:03:34.317420 15112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2: Bootstrap starting.
Sep 27 22:03:34 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:03:34.322384 15112 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2: Neither blocks nor log segments found. Creating new log.
I20260927 22:03:34.323971 15112 log.cc:824] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:03:34.328649 15112 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2: No bootstrap required, opened a new log
I20260927 22:03:34.331641 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:34.304262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.3.159.62:37927)
0927 22:03:34.305207 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:34.305220 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:34.305272 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:03:34.307340 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:34.307367 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:34.307896 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:34.308278 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:03:34.310406 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:34.312382 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:34.314589 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:34.315449 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:34.315662 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:34.321344 (+ 5682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:03:34.321357 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:03:34.321461 (+ 104us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:03:34.321549 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:03:34.326088 (+ 4539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:34.327391 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:34.327400 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:34.327408 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:34.327637 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:34.328236 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:34.328240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:34.328242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:34.328999 (+ 757us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:03:34.329355 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:34.329976 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 22:03:34.330465 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":435,"threads_started":1}
I20260927 22:03:34.339470 15112 raft_consensus.cc:348] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08523bda01e472a9232da5b4e586fc2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37927 } }
I20260927 22:03:34.340015 15112 raft_consensus.cc:374] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:03:34.340143 15112 raft_consensus.cc:729] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a08523bda01e472a9232da5b4e586fc2, State: Initialized, Role: FOLLOWER
I20260927 22:03:34.340862 15112 consensus_queue.cc:260] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08523bda01e472a9232da5b4e586fc2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37927 } }
I20260927 22:03:34.341162 15112 raft_consensus.cc:388] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:03:34.341325 15112 raft_consensus.cc:482] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:03:34.341555 15112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:03:34.345544 15112 raft_consensus.cc:504] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08523bda01e472a9232da5b4e586fc2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37927 } }
I20260927 22:03:34.346274 15112 leader_election.cc:302] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a08523bda01e472a9232da5b4e586fc2; no voters:
I20260927 22:03:34.347002 15112 leader_election.cc:288] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:03:34.347244 15117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:03:34.348198 15117 raft_consensus.cc:686] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [term 1 LEADER]: Becoming Leader. State: Replica: a08523bda01e472a9232da5b4e586fc2, State: Running, Role: LEADER
I20260927 22:03:34.348941 15117 consensus_queue.cc:237] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08523bda01e472a9232da5b4e586fc2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37927 } }
I20260927 22:03:34.349745 15112 sys_catalog.cc:564] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:03:34.352207 15119 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a08523bda01e472a9232da5b4e586fc2. Latest consensus state: current_term: 1 leader_uuid: "a08523bda01e472a9232da5b4e586fc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08523bda01e472a9232da5b4e586fc2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37927 } } }
I20260927 22:03:34.352682 15119 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:03:34.352461 15118 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a08523bda01e472a9232da5b4e586fc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08523bda01e472a9232da5b4e586fc2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37927 } } }
I20260927 22:03:34.352913 15118 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:03:34.356117 15124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:03:34.363273 15124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:03:34.367321 15112 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:03:34.387794 15124 catalog_manager.cc:1347] Generated new cluster ID: efd45dd5b1f34c299305dbd5319de21d
I20260927 22:03:34.388108 15124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:03:34.425498 15124 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:03:34.428115 15124 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:03:34.444525 15124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2: Generated new TSK 0
I20260927 22:03:34.446198 15124 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:03:35.286042 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:34629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37927
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:03:35 dist-test-slave-sg4j krb5kdc[14601](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790546615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:03:35.760450 15157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:03:35.761077 15157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:03:35.761269 15157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:03:35.774981 15157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:03:35.775213 15157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:03:35.775503 15157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:03:35.793898 15157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.159.20:34629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:03:35.796298 15157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:03:35.798682 15157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:03:35.822603 15168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:03:35.822643 15167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:03:35.831875 15170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:03:35.832906 15157 server_base.cc:1034] running on GCE node
I20260927 22:03:35.834136 15157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:03:35.835973 15157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:03:35.839458 15157 hybrid_clock.cc:648] HybridClock initialized: now 1790546615839336 us; error 93 us; skew 500 ppm
I20260927 22:03:35.840036 15157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:03:35 dist-test-slave-sg4j krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:03:35.845973 15157 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:03:35.850291 15157 webserver.cc:460] Webserver started at http://127.3.159.1:43905/ using document root <none> and password file <none>
I20260927 22:03:35.851722 15157 fs_manager.cc:359] Metadata directory not provided
I20260927 22:03:35.851927 15157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:03:35.852430 15157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:03:35 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:03:35.877836 15157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "718aa10730174e30b6cc0e6bb8599198"
format_stamp: "Formatted at 2026-09-27 22:03:35 on dist-test-slave-sg4j"
server_key: "9ee044a3a0e9295d8682de7c9b8f3a4b"
server_key_iv: "b97e38708b7f2eb3ae89c6759b5034f9"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:35.879088 15157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "718aa10730174e30b6cc0e6bb8599198"
format_stamp: "Formatted at 2026-09-27 22:03:35 on dist-test-slave-sg4j"
server_key: "9ee044a3a0e9295d8682de7c9b8f3a4b"
server_key_iv: "b97e38708b7f2eb3ae89c6759b5034f9"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:35.900839 15157 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.008s sys 0.012s
I20260927 22:03:35.926391 15177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:03:35.929416 15157 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.005s
I20260927 22:03:35.929689 15157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "718aa10730174e30b6cc0e6bb8599198"
format_stamp: "Formatted at 2026-09-27 22:03:35 on dist-test-slave-sg4j"
server_key: "9ee044a3a0e9295d8682de7c9b8f3a4b"
server_key_iv: "b97e38708b7f2eb3ae89c6759b5034f9"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:35.930018 15157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:03:35.974176 15157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:03:35.980060 15157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:03:35.980446 15157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:03:35.984899 15157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:03:35.995206 15157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:03:35.995354 15157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260927 22:03:35.995512 15157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:03:35.995581 15157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:03:36 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:03:36.649369 15289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:40461 every 8 connection(s)
I20260927 22:03:36.649760 15157 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:40461
I20260927 22:03:36.651612 15157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:03:36.653837 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 15157
I20260927 22:03:36.654131 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:03:36.654990 3708 external_mini_cluster.cc:1428] Setting key b4ca6e898ac30377aca8f456b1a51061
Sep 27 22:03:36 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:03:36.721781 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:36.661739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37927 (local address 127.3.159.1:50915)
0927 22:03:36.689283 (+ 27544us) negotiation.cc:107] Waiting for socket to connect
0927 22:03:36.689348 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 22:03:36.692547 (+ 3199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:03:36.694921 (+ 2374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:03:36.694935 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:03:36.695629 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:03:36.696810 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:36.696846 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:36.698624 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:36.698630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:36.700839 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:36.700864 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:36.702441 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:36.702449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:36.702765 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:36.705654 (+ 2889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:03:36.705689 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:03:36.710573 (+ 4884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:03:36.715478 (+ 4905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:03:36.715491 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:03:36.715513 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:03:36.716281 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:03:36.716808 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:03:36.716813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:03:36.716818 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:03:36.717126 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:03:36.718157 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:03:36.718169 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:03:36.718720 (+ 551us) client_negotiation.cc:769] Sending connection context
0927 22:03:36.719498 (+ 778us) client_negotiation.cc:240] Negotiation successful
0927 22:03:36.720131 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27180,"thread_start_us":31832,"threads_started":1}
I20260927 22:03:36.721990 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:36.662421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:50915 (local address 127.3.159.62:37927)
0927 22:03:36.688514 (+ 26093us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:36.688525 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:36.693137 (+ 4612us) server_negotiation.cc:407] Connection header received
0927 22:03:36.693220 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:36.693225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:36.693327 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:36.693510 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:03:36.697094 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:36.698337 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:36.701251 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:36.701924 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:36.702040 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:36.710969 (+ 8929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:03:36.710978 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:03:36.710983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:03:36.711051 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:03:36.715215 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:36.716463 (+ 1248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:36.716470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:36.716474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:36.716655 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:36.717284 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:36.717290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:36.717294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:36.717735 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:03:36.717885 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:36.721169 (+ 3284us) server_negotiation.cc:299] Negotiation successful
0927 22:03:36.721560 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25855,"thread_start_us":26139,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:03:36.733529 15290 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37927
I20260927 22:03:36.734018 15290 heartbeater.cc:464] Registering TS with master...
I20260927 22:03:36.735049 15290 heartbeater.cc:511] Master 127.3.159.62:37927 requested a full tablet report, sending...
I20260927 22:03:36.738134 15077 ts_manager.cc:194] Registered new tserver with Master: 718aa10730174e30b6cc0e6bb8599198 (127.3.159.1:40461)
I20260927 22:03:36.740252 15077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:50915
Sep 27 22:03:36 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
W20260927 22:03:36.797127 3708 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.159.2: real 0.138s user 0.001s sys 0.019s
I20260927 22:03:36.798219 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:34629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37927
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:03:37.378343 15297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:03:37.378809 15297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:03:37.378994 15297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:03:37.388314 15297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:03:37.388502 15297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:03:37.388708 15297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:03:37.400231 15297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.159.20:34629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:03:37.402047 15297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:03:37.403780 15297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:03:37.422612 15311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:03:37.423417 15310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:03:37.423507 15313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:03:37.424110 15297 server_base.cc:1034] running on GCE node
I20260927 22:03:37.425208 15297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:03:37.426801 15297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:03:37.428133 15297 hybrid_clock.cc:648] HybridClock initialized: now 1790546617428049 us; error 75 us; skew 500 ppm
I20260927 22:03:37.428666 15297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:03:37 dist-test-slave-sg4j krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:03:37.433864 15297 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:03:37.436518 15297 webserver.cc:460] Webserver started at http://127.3.159.2:36117/ using document root <none> and password file <none>
I20260927 22:03:37.437682 15297 fs_manager.cc:359] Metadata directory not provided
I20260927 22:03:37.437876 15297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:03:37.438328 15297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:03:37 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:03:37.465073 15297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "f1488e88cb9644d4a86c3ad402499211"
format_stamp: "Formatted at 2026-09-27 22:03:37 on dist-test-slave-sg4j"
server_key: "bcf479a1cb74bcefa5103922d8416c79"
server_key_iv: "480b560ce5bc33fcbc6473e77725d879"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:37.466773 15297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "f1488e88cb9644d4a86c3ad402499211"
format_stamp: "Formatted at 2026-09-27 22:03:37 on dist-test-slave-sg4j"
server_key: "bcf479a1cb74bcefa5103922d8416c79"
server_key_iv: "480b560ce5bc33fcbc6473e77725d879"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:37.476905 15297 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260927 22:03:37.501639 15320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:03:37.504500 15297 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260927 22:03:37.504801 15297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "f1488e88cb9644d4a86c3ad402499211"
format_stamp: "Formatted at 2026-09-27 22:03:37 on dist-test-slave-sg4j"
server_key: "bcf479a1cb74bcefa5103922d8416c79"
server_key_iv: "480b560ce5bc33fcbc6473e77725d879"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:37.505162 15297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:03:37.550093 15297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:03:37.557562 15297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:03:37.558017 15297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:03:37.559696 15297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:03:37.562337 15297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:03:37.562467 15297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:03:37.562619 15297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:03:37.562705 15297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:03:37.615408 15297 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:34501
I20260927 22:03:37.615455 15432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:34501 every 8 connection(s)
I20260927 22:03:37.617717 15297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:03:37.625631 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 15297
I20260927 22:03:37.625841 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:03:37.626314 3708 external_mini_cluster.cc:1428] Setting key 96de538be15e96c58f3a1308f26b4653
Sep 27 22:03:37 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546617, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:03:37.653257 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:37.621681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:60797 (local address 127.3.159.62:37927)
0927 22:03:37.622315 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:37.622323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:37.623856 (+ 1533us) server_negotiation.cc:407] Connection header received
0927 22:03:37.627330 (+ 3474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:37.627343 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:37.627496 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:37.627850 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:03:37.633201 (+ 5351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:37.634653 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:37.637435 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:37.637874 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:37.638034 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:37.642177 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:03:37.642188 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:03:37.642194 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:03:37.642275 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:03:37.647708 (+ 5433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:37.649305 (+ 1597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:37.649313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:37.649318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:37.649565 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:37.650367 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:37.650373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:37.650378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:37.650886 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:03:37.651039 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:37.652412 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0927 22:03:37.652768 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":308,"threads_started":1}
I20260927 22:03:37.656286 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:37.622344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37927 (local address 127.3.159.2:60797)
0927 22:03:37.623539 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0927 22:03:37.623632 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0927 22:03:37.626673 (+ 3041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:03:37.628211 (+ 1538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:03:37.628223 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:03:37.628907 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:03:37.629879 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:37.629908 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:37.634916 (+ 5008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:37.634932 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:37.637179 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:37.637212 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:37.638139 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:37.638147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:37.638446 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:37.639228 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:03:37.639253 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:03:37.641734 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:03:37.648056 (+ 6322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:03:37.648071 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:03:37.648102 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:03:37.649037 (+ 935us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:03:37.649850 (+ 813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:03:37.649855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:03:37.649859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:03:37.650203 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:03:37.651174 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:03:37.651185 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:03:37.651793 (+ 608us) client_negotiation.cc:769] Sending connection context
0927 22:03:37.653581 (+ 1788us) client_negotiation.cc:240] Negotiation successful
0927 22:03:37.654377 (+ 796us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":413,"threads_started":1}
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
I20260927 22:03:37.658661 15433 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37927
I20260927 22:03:37.659135 15433 heartbeater.cc:464] Registering TS with master...
I20260927 22:03:37.660156 15433 heartbeater.cc:511] Master 127.3.159.62:37927 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
I20260927 22:03:37.662534 15077 ts_manager.cc:194] Registered new tserver with Master: f1488e88cb9644d4a86c3ad402499211 (127.3.159.2:34501)
I20260927 22:03:37.663837 15077 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:60797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:03:37.694600 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:34629/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37927
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:45647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:03:37.744536 15290 heartbeater.cc:503] Master 127.3.159.62:37927 was elected leader, sending a full tablet report...
W20260927 22:03:38.156697 15439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:03:38.157115 15439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:03:38.157315 15439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:03:38.166059 15439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:03:38.166276 15439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:03:38.166543 15439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:03:38.183750 15439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:45647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.159.20:34629/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:03:38.185356 15439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:03:38.186986 15439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:03:38.204290 15445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:03:38.204319 15444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:03:38.205868 15447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:03:38.205749 15439 server_base.cc:1034] running on GCE node
I20260927 22:03:38.206667 15439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:03:38.207931 15439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:03:38.209244 15439 hybrid_clock.cc:648] HybridClock initialized: now 1790546618209180 us; error 52 us; skew 500 ppm
I20260927 22:03:38.209785 15439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:03:38 dist-test-slave-sg4j krb5kdc[14601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:03:38.214090 15439 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:03:38.216048 15439 webserver.cc:460] Webserver started at http://127.3.159.3:38133/ using document root <none> and password file <none>
I20260927 22:03:38.216813 15439 fs_manager.cc:359] Metadata directory not provided
I20260927 22:03:38.216984 15439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:03:38.217361 15439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:03:38 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:03:38.236632 15439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "8d75428c5ed04d7e9e4c25712b216326"
format_stamp: "Formatted at 2026-09-27 22:03:38 on dist-test-slave-sg4j"
server_key: "0beae95636bbc9b5743a7a9127bbc6ee"
server_key_iv: "582e49d51b4b5842e8cdfcd6a053cb1b"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:38.237844 15439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "8d75428c5ed04d7e9e4c25712b216326"
format_stamp: "Formatted at 2026-09-27 22:03:38 on dist-test-slave-sg4j"
server_key: "0beae95636bbc9b5743a7a9127bbc6ee"
server_key_iv: "582e49d51b4b5842e8cdfcd6a053cb1b"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:38.243909 15439 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:03:38.267220 15454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:03:38.269806 15439 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.001s
I20260927 22:03:38.270102 15439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "8d75428c5ed04d7e9e4c25712b216326"
format_stamp: "Formatted at 2026-09-27 22:03:38 on dist-test-slave-sg4j"
server_key: "0beae95636bbc9b5743a7a9127bbc6ee"
server_key_iv: "582e49d51b4b5842e8cdfcd6a053cb1b"
server_key_version: "kuduclusterkey@0"
I20260927 22:03:38.270433 15439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:03:38.302959 15439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:03:38.309165 15439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:03:38.309594 15439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:03:38.311120 15439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:03:38.313341 15439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:03:38.313463 15439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 22:03:38.313601 15439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:03:38.313671 15439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:03:38.376263 15439 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:44667
I20260927 22:03:38.376473 15566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:44667 every 8 connection(s)
I20260927 22:03:38.378461 15439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:03:38.381664 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 15439
I20260927 22:03:38.381886 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:03:38.382449 3708 external_mini_cluster.cc:1428] Setting key 21c0c37c1c91e39f5e1050bb0d91ecc4
Sep 27 22:03:38 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:03:38.404668 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.382030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:51277 (local address 127.3.159.62:37927)
0927 22:03:38.382689 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:38.382697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:38.384136 (+ 1439us) server_negotiation.cc:407] Connection header received
0927 22:03:38.386436 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:38.386440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:38.386540 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:38.386652 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:03:38.388537 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.389518 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.391606 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.392090 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.392224 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.396987 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:03:38.396995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:03:38.396999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:03:38.397055 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:03:38.400596 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:38.401879 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:38.401884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:38.401887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:38.402087 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:38.402663 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:38.402667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:38.402669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:38.403002 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:03:38.403139 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:38.404186 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0927 22:03:38.404390 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":320,"threads_started":1}
I20260927 22:03:38.406118 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.382865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37927 (local address 127.3.159.3:51277)
0927 22:03:38.383924 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0927 22:03:38.383981 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:03:38.386054 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:03:38.386977 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:03:38.386987 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:03:38.387539 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:03:38.388277 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.388297 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.389750 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.389755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.391382 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.391406 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.392358 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.392363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.392628 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.393593 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:03:38.393635 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:03:38.396633 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:03:38.400837 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:03:38.400848 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:03:38.400869 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:03:38.401687 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:03:38.402264 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:03:38.402268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:03:38.402271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:03:38.402526 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:03:38.403213 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:03:38.403221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:03:38.403719 (+ 498us) client_negotiation.cc:769] Sending connection context
0927 22:03:38.404001 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 22:03:38.404512 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":463,"threads_started":1}
I20260927 22:03:38.408226 15567 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37927
I20260927 22:03:38.408563 15567 heartbeater.cc:464] Registering TS with master...
I20260927 22:03:38.409325 15567 heartbeater.cc:511] Master 127.3.159.62:37927 requested a full tablet report, sending...
I20260927 22:03:38.411369 15076 ts_manager.cc:194] Registered new tserver with Master: 8d75428c5ed04d7e9e4c25712b216326 (127.3.159.3:44667)
I20260927 22:03:38.412683 15076 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:51277
I20260927 22:03:38.412777 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:03:38.436468 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.418735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.3.159.62:37927)
0927 22:03:38.418939 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:38.418944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:38.419490 (+ 546us) server_negotiation.cc:407] Connection header received
0927 22:03:38.419627 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:38.419632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:38.419718 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:38.419826 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:03:38.422737 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.423612 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.425419 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.426037 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.426203 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.429464 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:03:38.429479 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:03:38.429483 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:03:38.429532 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:03:38.432606 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:38.433753 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:38.433759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:38.433762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:38.433981 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:38.434489 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:38.434493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:38.434495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:38.434890 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:03:38.435150 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:38.435762 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 22:03:38.436075 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:03:38.442523 15076 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:03:38.451119 15076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:03:38.497601 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.469933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:44667 (local address 127.0.0.1:46528)
0927 22:03:38.483930 (+ 13997us) negotiation.cc:107] Waiting for socket to connect
0927 22:03:38.483961 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 22:03:38.484148 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:03:38.488442 (+ 4294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:03:38.488447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:03:38.488468 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:03:38.489083 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.489092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.491306 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.491313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.493964 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.493982 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.496001 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.496008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.496204 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.496257 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 22:03:38.496866 (+ 609us) client_negotiation.cc:240] Negotiation successful
0927 22:03:38.497096 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13792,"mutex_wait_us":130,"thread_start_us":12069,"threads_started":1}
I20260927 22:03:38.497772 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.470050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.3.159.3:44667)
0927 22:03:38.487776 (+ 17726us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:38.487787 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:38.487822 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:03:38.487923 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:38.487930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:38.488149 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:38.488348 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:03:38.489339 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.491095 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.494182 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.495456 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.495603 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.495854 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:38.497005 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0927 22:03:38.497271 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17511,"thread_start_us":17259,"threads_started":1}
I20260927 22:03:38.498615 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.469489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:40461 (local address 127.0.0.1:36336)
0927 22:03:38.479639 (+ 10150us) negotiation.cc:107] Waiting for socket to connect
0927 22:03:38.479717 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 22:03:38.480050 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:03:38.488160 (+ 8110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:03:38.488167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:03:38.488202 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:03:38.488764 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.488782 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.491305 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.491311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.494786 (+ 3475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.494804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.497665 (+ 2861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.497671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.497865 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.497896 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 22:03:38.497997 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 22:03:38.498158 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9670,"mutex_wait_us":141,"thread_start_us":15802,"threads_started":1}
I20260927 22:03:38.499158 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.469737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.3.159.1:40461)
0927 22:03:38.485006 (+ 15269us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:38.485024 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:38.485075 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:03:38.487612 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:38.487626 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:38.487891 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:38.488134 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:03:38.489096 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.491104 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.495033 (+ 3929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.497093 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.497248 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.497507 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:38.498386 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 22:03:38.498606 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14984,"thread_start_us":14765,"threads_started":1}
I20260927 22:03:38.499514 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.474686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:34501 (local address 127.0.0.1:44996)
0927 22:03:38.487068 (+ 12382us) negotiation.cc:107] Waiting for socket to connect
0927 22:03:38.487104 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 22:03:38.487356 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:03:38.488655 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:03:38.488664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:03:38.488765 (+ 101us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:03:38.489464 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.489478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.491433 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.491441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.495558 (+ 4117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.495574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.498585 (+ 3011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.498593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.498801 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.498838 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:03:38.498942 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:03:38.499117 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12161,"mutex_wait_us":867,"thread_start_us":11935,"threads_started":1}
I20260927 22:03:38.502869 15502 tablet_service.cc:1467] Processing CreateTablet for tablet 77a186aaf8034ef5bc9a7222606a0bc7 (DEFAULT_TABLE table=test-table [id=88831f44b75d4fb5bf7cda9e06cd121b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:03:38.503302 15225 tablet_service.cc:1467] Processing CreateTablet for tablet 77a186aaf8034ef5bc9a7222606a0bc7 (DEFAULT_TABLE table=test-table [id=88831f44b75d4fb5bf7cda9e06cd121b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:03:38.504185 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.475406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.3.159.2:34501)
0927 22:03:38.486398 (+ 10992us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:38.486413 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:38.487296 (+ 883us) server_negotiation.cc:407] Connection header received
0927 22:03:38.487590 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:38.487602 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:38.487999 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:38.488303 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:03:38.489744 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.491308 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.495801 (+ 4493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.497908 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.498090 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.498345 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:38.500248 (+ 1903us) server_negotiation.cc:299] Negotiation successful
0927 22:03:38.503780 (+ 3532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10688,"thread_start_us":10500,"threads_started":1}
I20260927 22:03:38.504518 15502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a186aaf8034ef5bc9a7222606a0bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:03:38.506095 15225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a186aaf8034ef5bc9a7222606a0bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:03:38.508376 15368 tablet_service.cc:1467] Processing CreateTablet for tablet 77a186aaf8034ef5bc9a7222606a0bc7 (DEFAULT_TABLE table=test-table [id=88831f44b75d4fb5bf7cda9e06cd121b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:03:38.510010 15368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a186aaf8034ef5bc9a7222606a0bc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:03:38.520588 15586 tablet_bootstrap.cc:492] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: Bootstrap starting.
I20260927 22:03:38.524674 15586 tablet_bootstrap.cc:654] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: Neither blocks nor log segments found. Creating new log.
I20260927 22:03:38.526162 15586 log.cc:824] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: Log is configured to *not* fsync() on all Append() calls
I20260927 22:03:38.528486 15589 tablet_bootstrap.cc:492] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: Bootstrap starting.
I20260927 22:03:38.528867 15587 tablet_bootstrap.cc:492] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: Bootstrap starting.
I20260927 22:03:38.532358 15586 tablet_bootstrap.cc:492] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: No bootstrap required, opened a new log
I20260927 22:03:38.532743 15586 ts_tablet_manager.cc:1397] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260927 22:03:38.534090 15587 tablet_bootstrap.cc:654] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: Neither blocks nor log segments found. Creating new log.
I20260927 22:03:38.534089 15589 tablet_bootstrap.cc:654] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: Neither blocks nor log segments found. Creating new log.
I20260927 22:03:38.535616 15587 log.cc:824] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: Log is configured to *not* fsync() on all Append() calls
I20260927 22:03:38.535610 15589 log.cc:824] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: Log is configured to *not* fsync() on all Append() calls
I20260927 22:03:38.540709 15589 tablet_bootstrap.cc:492] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: No bootstrap required, opened a new log
I20260927 22:03:38.541076 15589 ts_tablet_manager.cc:1397] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260927 22:03:38.541802 15587 tablet_bootstrap.cc:492] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: No bootstrap required, opened a new log
I20260927 22:03:38.542073 15587 ts_tablet_manager.cc:1397] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260927 22:03:38.542850 15586 raft_consensus.cc:348] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718aa10730174e30b6cc0e6bb8599198" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 40461 } } peers { permanent_uuid: "8d75428c5ed04d7e9e4c25712b216326" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44667 } } peers { permanent_uuid: "f1488e88cb9644d4a86c3ad402499211" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34501 } }
I20260927 22:03:38.543336 15586 raft_consensus.cc:374] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:03:38.543442 15586 raft_consensus.cc:729] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718aa10730174e30b6cc0e6bb8599198, State: Initialized, Role: FOLLOWER
I20260927 22:03:38.544302 15586 consensus_queue.cc:260] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718aa10730174e30b6cc0e6bb8599198" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 40461 } } peers { permanent_uuid: "8d75428c5ed04d7e9e4c25712b216326" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44667 } } peers { permanent_uuid: "f1488e88cb9644d4a86c3ad402499211" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34501 } }
I20260927 22:03:38.546706 15586 ts_tablet_manager.cc:1428] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260927 22:03:38.553840 15589 raft_consensus.cc:348] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718aa10730174e30b6cc0e6bb8599198" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 40461 } } peers { permanent_uuid: "8d75428c5ed04d7e9e4c25712b216326" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44667 } } peers { permanent_uuid: "f1488e88cb9644d4a86c3ad402499211" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34501 } }
I20260927 22:03:38.554531 15589 raft_consensus.cc:374] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:03:38.554661 15589 raft_consensus.cc:729] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1488e88cb9644d4a86c3ad402499211, State: Initialized, Role: FOLLOWER
I20260927 22:03:38.554771 15587 raft_consensus.cc:348] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718aa10730174e30b6cc0e6bb8599198" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 40461 } } peers { permanent_uuid: "8d75428c5ed04d7e9e4c25712b216326" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44667 } } peers { permanent_uuid: "f1488e88cb9644d4a86c3ad402499211" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34501 } }
I20260927 22:03:38.555227 15587 raft_consensus.cc:374] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:03:38.555338 15587 raft_consensus.cc:729] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d75428c5ed04d7e9e4c25712b216326, State: Initialized, Role: FOLLOWER
I20260927 22:03:38.555543 15589 consensus_queue.cc:260] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718aa10730174e30b6cc0e6bb8599198" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 40461 } } peers { permanent_uuid: "8d75428c5ed04d7e9e4c25712b216326" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44667 } } peers { permanent_uuid: "f1488e88cb9644d4a86c3ad402499211" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34501 } }
I20260927 22:03:38.555949 15587 consensus_queue.cc:260] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718aa10730174e30b6cc0e6bb8599198" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 40461 } } peers { permanent_uuid: "8d75428c5ed04d7e9e4c25712b216326" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44667 } } peers { permanent_uuid: "f1488e88cb9644d4a86c3ad402499211" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34501 } }
I20260927 22:03:38.557137 15433 heartbeater.cc:503] Master 127.3.159.62:37927 was elected leader, sending a full tablet report...
I20260927 22:03:38.557411 15567 heartbeater.cc:503] Master 127.3.159.62:37927 was elected leader, sending a full tablet report...
I20260927 22:03:38.558163 15587 ts_tablet_manager.cc:1428] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260927 22:03:38.559244 15589 ts_tablet_manager.cc:1428] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
W20260927 22:03:38.621186 15434 tablet.cc:2369] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:03:38.630726 15568 tablet.cc:2369] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:03:38.689217 15294 tablet.cc:2369] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:03:38.733242 15594 raft_consensus.cc:482] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:03:38.733745 15594 raft_consensus.cc:504] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718aa10730174e30b6cc0e6bb8599198" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 40461 } } peers { permanent_uuid: "8d75428c5ed04d7e9e4c25712b216326" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44667 } } peers { permanent_uuid: "f1488e88cb9644d4a86c3ad402499211" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34501 } }
I20260927 22:03:38.735533 15594 leader_election.cc:288] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 718aa10730174e30b6cc0e6bb8599198 (127.3.159.1:40461), f1488e88cb9644d4a86c3ad402499211 (127.3.159.2:34501)
I20260927 22:03:38.742249 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.736038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:40461 (local address 127.3.159.3:59221)
0927 22:03:38.736400 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 22:03:38.736442 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:03:38.736648 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:03:38.737092 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:03:38.737098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:03:38.737133 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:03:38.737548 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.737565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.738719 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.738726 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.740251 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.740265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.741468 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.741474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.741661 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.741687 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:03:38.741798 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:03:38.741947 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":26}
I20260927 22:03:38.742223 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.736281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:59221 (local address 127.3.159.1:40461)
0927 22:03:38.736538 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:38.736547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:38.736621 (+ 74us) server_negotiation.cc:407] Connection header received
0927 22:03:38.736855 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:38.736862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:38.736957 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:38.737076 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:03:38.737748 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.738549 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.740438 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.741242 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.741390 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.741533 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:38.741828 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 22:03:38.741969 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 22:03:38.743698 15245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a186aaf8034ef5bc9a7222606a0bc7" candidate_uuid: "8d75428c5ed04d7e9e4c25712b216326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718aa10730174e30b6cc0e6bb8599198" is_pre_election: true
I20260927 22:03:38.744346 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.736184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:45673 (local address 127.3.159.2:34501)
0927 22:03:38.736409 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:38.736416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:38.737608 (+ 1192us) server_negotiation.cc:407] Connection header received
0927 22:03:38.737803 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:38.737812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:38.737904 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:38.737963 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:03:38.738738 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.739492 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.741407 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.743263 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.743425 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.743545 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:38.743882 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 22:03:38.744061 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:03:38.744345 15595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.736281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:34501 (local address 127.3.159.3:45673)
0927 22:03:38.737305 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0927 22:03:38.737325 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:03:38.737609 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:03:38.738199 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:03:38.738204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:03:38.738235 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:03:38.738568 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.738576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.739677 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.739682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.741226 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:03:38.741238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.743526 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.743532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:03:38.743668 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.743694 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 22:03:38.743809 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 22:03:38.743934 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":48,"thread_start_us":580,"threads_started":1}
I20260927 22:03:38.744501 15245 raft_consensus.cc:2454] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d75428c5ed04d7e9e4c25712b216326 in term 0.
I20260927 22:03:38.745488 15456 leader_election.cc:302] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718aa10730174e30b6cc0e6bb8599198, 8d75428c5ed04d7e9e4c25712b216326; no voters:
I20260927 22:03:38.745801 15388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a186aaf8034ef5bc9a7222606a0bc7" candidate_uuid: "8d75428c5ed04d7e9e4c25712b216326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1488e88cb9644d4a86c3ad402499211" is_pre_election: true
I20260927 22:03:38.746060 15594 raft_consensus.cc:2781] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:03:38.746296 15594 raft_consensus.cc:482] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:03:38.746410 15594 raft_consensus.cc:3037] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:03:38.746492 15388 raft_consensus.cc:2454] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d75428c5ed04d7e9e4c25712b216326 in term 0.
I20260927 22:03:38.748890 15594 raft_consensus.cc:504] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718aa10730174e30b6cc0e6bb8599198" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 40461 } } peers { permanent_uuid: "8d75428c5ed04d7e9e4c25712b216326" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44667 } } peers { permanent_uuid: "f1488e88cb9644d4a86c3ad402499211" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34501 } }
I20260927 22:03:38.749677 15594 leader_election.cc:288] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [CANDIDATE]: Term 1 election: Requested vote from peers 718aa10730174e30b6cc0e6bb8599198 (127.3.159.1:40461), f1488e88cb9644d4a86c3ad402499211 (127.3.159.2:34501)
I20260927 22:03:38.750451 15388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a186aaf8034ef5bc9a7222606a0bc7" candidate_uuid: "8d75428c5ed04d7e9e4c25712b216326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1488e88cb9644d4a86c3ad402499211"
I20260927 22:03:38.750447 15245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a186aaf8034ef5bc9a7222606a0bc7" candidate_uuid: "8d75428c5ed04d7e9e4c25712b216326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "718aa10730174e30b6cc0e6bb8599198"
I20260927 22:03:38.750707 15245 raft_consensus.cc:3037] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:03:38.750708 15388 raft_consensus.cc:3037] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:03:38.754066 15245 raft_consensus.cc:2454] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d75428c5ed04d7e9e4c25712b216326 in term 1.
I20260927 22:03:38.754066 15388 raft_consensus.cc:2454] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d75428c5ed04d7e9e4c25712b216326 in term 1.
I20260927 22:03:38.754665 15456 leader_election.cc:302] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 718aa10730174e30b6cc0e6bb8599198, 8d75428c5ed04d7e9e4c25712b216326; no voters:
I20260927 22:03:38.755031 15594 raft_consensus.cc:2781] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:03:38.755548 15594 raft_consensus.cc:686] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 1 LEADER]: Becoming Leader. State: Replica: 8d75428c5ed04d7e9e4c25712b216326, State: Running, Role: LEADER
I20260927 22:03:38.756197 15594 consensus_queue.cc:237] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718aa10730174e30b6cc0e6bb8599198" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 40461 } } peers { permanent_uuid: "8d75428c5ed04d7e9e4c25712b216326" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44667 } } peers { permanent_uuid: "f1488e88cb9644d4a86c3ad402499211" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34501 } }
I20260927 22:03:38.761763 15077 catalog_manager.cc:5495] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d75428c5ed04d7e9e4c25712b216326 (127.3.159.3). New cstate: current_term: 1 leader_uuid: "8d75428c5ed04d7e9e4c25712b216326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "718aa10730174e30b6cc0e6bb8599198" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d75428c5ed04d7e9e4c25712b216326" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1488e88cb9644d4a86c3ad402499211" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34501 } health_report { overall_health: UNKNOWN } } }
I20260927 22:03:38.804301 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.797405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.3.159.3:44667)
0927 22:03:38.797632 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:38.797638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:38.797664 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:03:38.797781 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:38.797786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:38.797925 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:38.798091 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:03:38.798790 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.799659 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.801527 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.801960 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.802136 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.802706 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:03:38.803500 (+ 794us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:03:38.803646 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:38.803826 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 22:03:38.803953 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:03:38.814869 15388 raft_consensus.cc:1261] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211 [term 1 FOLLOWER]: Refusing update from remote peer 8d75428c5ed04d7e9e4c25712b216326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:03:38.815007 15245 raft_consensus.cc:1261] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198 [term 1 FOLLOWER]: Refusing update from remote peer 8d75428c5ed04d7e9e4c25712b216326: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:03:38.816087 15594 consensus_queue.cc:1035] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "718aa10730174e30b6cc0e6bb8599198" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:03:38.816466 15596 consensus_queue.cc:1035] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1488e88cb9644d4a86c3ad402499211" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 34501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:03:38.833832 15600 mvcc.cc:205] Tried to move back new op lower bound from 7334078950651101184 to 7334078950429720576. Current Snapshot: MvccSnapshot[applied={T|T < 7334078950651101184}]
I20260927 22:03:38.850190 15076 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:03:38.850610 15076 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:03:38.855378 15076 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a08523bda01e472a9232da5b4e586fc2: Sending DeleteTablet for 3 replicas of tablet 77a186aaf8034ef5bc9a7222606a0bc7
I20260927 22:03:38.857183 15225 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a186aaf8034ef5bc9a7222606a0bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:03:38 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:36336
I20260927 22:03:38.857189 15502 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a186aaf8034ef5bc9a7222606a0bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:03:38 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:46528
I20260927 22:03:38.858251 15368 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a186aaf8034ef5bc9a7222606a0bc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:03:38 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:44996
I20260927 22:03:38.858860 15610 tablet_replica.cc:330] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: stopping tablet replica
I20260927 22:03:38.859411 15610 raft_consensus.cc:2229] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:03:38.859763 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 15157
I20260927 22:03:38.860131 15610 raft_consensus.cc:2258] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:03:38.871033 15610 ts_tablet_manager.cc:1905] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:03:38.873907 15609 tablet_replica.cc:330] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: stopping tablet replica
I20260927 22:03:38.874351 15611 tablet_replica.cc:330] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: stopping tablet replica
I20260927 22:03:38.874464 15609 raft_consensus.cc:2229] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:03:38.875070 15609 raft_consensus.cc:2258] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:03:38.875057 15611 raft_consensus.cc:2229] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:03:38.875771 15611 raft_consensus.cc:2258] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:03:38.877763 15609 ts_tablet_manager.cc:1905] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:03:38.885155 15610 ts_tablet_manager.cc:1918] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:03:38.885530 15610 log.cc:1196] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/77a186aaf8034ef5bc9a7222606a0bc7
I20260927 22:03:38.886206 15610 ts_tablet_manager.cc:1939] T 77a186aaf8034ef5bc9a7222606a0bc7 P 8d75428c5ed04d7e9e4c25712b216326: Deleting consensus metadata
I20260927 22:03:38.887272 15609 ts_tablet_manager.cc:1918] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:03:38.887629 15609 log.cc:1196] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/77a186aaf8034ef5bc9a7222606a0bc7
I20260927 22:03:38.888315 15609 ts_tablet_manager.cc:1939] T 77a186aaf8034ef5bc9a7222606a0bc7 P 718aa10730174e30b6cc0e6bb8599198: Deleting consensus metadata
I20260927 22:03:38.888666 15064 catalog_manager.cc:4841] TS 8d75428c5ed04d7e9e4c25712b216326 (127.3.159.3:44667): tablet 77a186aaf8034ef5bc9a7222606a0bc7 (table test-table [id=88831f44b75d4fb5bf7cda9e06cd121b]) successfully deleted
Sep 27 22:03:38 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:03:38.890599 15611 ts_tablet_manager.cc:1905] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:03:38.893258 15062 catalog_manager.cc:4841] TS 718aa10730174e30b6cc0e6bb8599198 (127.3.159.1:40461): tablet 77a186aaf8034ef5bc9a7222606a0bc7 (table test-table [id=88831f44b75d4fb5bf7cda9e06cd121b]) successfully deleted
I20260927 22:03:38.898187 15611 ts_tablet_manager.cc:1918] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:03:38.898411 15611 log.cc:1196] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/77a186aaf8034ef5bc9a7222606a0bc7
I20260927 22:03:38.898913 15611 ts_tablet_manager.cc:1939] T 77a186aaf8034ef5bc9a7222606a0bc7 P f1488e88cb9644d4a86c3ad402499211: Deleting consensus metadata
I20260927 22:03:38.901022 15063 catalog_manager.cc:4841] TS f1488e88cb9644d4a86c3ad402499211 (127.3.159.2:34501): tablet 77a186aaf8034ef5bc9a7222606a0bc7 (table test-table [id=88831f44b75d4fb5bf7cda9e06cd121b]) successfully deleted
I20260927 22:03:38.901573 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:38.861792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.3.159.1:40461)
0927 22:03:38.862068 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:38.862079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:38.876747 (+ 14668us) server_negotiation.cc:407] Connection header received
0927 22:03:38.877084 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:38.877091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:38.877182 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:38.877257 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:03:38.879508 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.880762 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.883217 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:38.884013 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:38.884152 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:38.891806 (+ 7654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:03:38.891825 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:03:38.891850 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:03:38.891923 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:03:38.897057 (+ 5134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:38.898514 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:38.898529 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:38.898533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:38.898777 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:38.899398 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:38.899406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:38.899409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:38.900013 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:03:38.900372 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:38.900805 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 22:03:38.901064 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:03:38.902525 15285 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:03:39.006705 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 15157
I20260927 22:03:39.027300 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 15297
Sep 27 22:03:39 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:03:39.050510 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:39.028382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.3.159.2:34501)
0927 22:03:39.028638 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:39.028646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:39.028674 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:03:39.028763 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:39.028770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:39.028877 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:39.029035 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:03:39.030702 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:39.032354 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:39.034305 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:39.035089 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:39.035305 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:39.041363 (+ 6058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:03:39.041421 (+ 58us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:03:39.041455 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:03:39.041526 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:03:39.046334 (+ 4808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:39.047555 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:39.047567 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:39.047571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:39.047789 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:39.048276 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:39.048281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:39.048285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:39.048892 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:03:39.049247 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:39.049745 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 22:03:39.050043 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:03:39.051927 15428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:03:39.159044 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 15297
I20260927 22:03:39.184525 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 15439
Sep 27 22:03:39 dist-test-slave-sg4j krb5kdc[14601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:03:39.211465 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:03:39.185693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.3.159.3:44667)
0927 22:03:39.185970 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 22:03:39.185978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:03:39.186008 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:03:39.186103 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:03:39.186110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:03:39.186228 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:03:39.186390 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:03:39.188192 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:39.189636 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:39.191564 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:03:39.192404 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:03:39.192625 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:03:39.203424 (+ 10799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:03:39.203444 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:03:39.203462 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:03:39.203530 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:03:39.207655 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:39.208933 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:39.208941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:39.208943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:39.209166 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:03:39.209715 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:03:39.209719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:03:39.209722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:03:39.210120 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:03:39.210396 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 22:03:39.210877 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 22:03:39.211069 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 22:03:39.212462 15562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:03:39.350720 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 15439
I20260927 22:03:39.376794 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 15045
I20260927 22:03:39.378480 15107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:03:39.504292 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 15045
I20260927 22:03:39.531229 3708 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:03:39.878316 3708 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:03:39.878628 3708 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:03:40.226924 3708 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:03:40.227 UTC [14631] LOG: received smart shutdown request
2026-09-27 22:03:40.231 UTC [14631] LOG: background worker "logical replication launcher" (PID 14640) exited with exit code 1
2026-09-27 22:03:40.231 UTC [14635] LOG: shutting down
2026-09-27 22:03:40.447 UTC [14631] LOG: database system is shut down
2026-09-27T22:03:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (58444 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:03:40 dist-test-slave-sg4j krb5kdc[15625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:03:40 dist-test-slave-sg4j krb5kdc[15625](info): set up 2 sockets
Sep 27 22:03:40 dist-test-slave-sg4j krb5kdc[15625](info): commencing operation
krb5kdc: starting...
W20260927 22:03:41.730284 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:03:41 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:03:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:03:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.159.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.159.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/rangeradmin_127.3.159.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/rangeradmin_127.3.159.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.159.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.159.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/rangerlookup_127.3.159.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/rangerlookup_127.3.159.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.159.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.159.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/HTTP_127.3.159.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/HTTP_127.3.159.20@KRBTEST.COM.keytab.
I20260927 22:03:42.150126 3708 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIPxQZD/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/postgres -l logfile start
2026-09-27 22:03:45.353 UTC [15655] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 22:03:45.353 UTC [15655] LOG: listening on IPv4 address "127.3.159.20", port 36715
2026-09-27 22:03:45.359 UTC [15655] LOG: listening on Unix socket "/tmp/.s.PGSQL.36715"
2026-09-27 22:03:45.394 UTC [15658] LOG: database system was shut down at 2026-09-27 22:03:43 UTC
2026-09-27 22:03:45.400 UTC [15655] LOG: database system is ready to accept connections
I20260927 22:03:46.393831 3708 mini_postgres.cc:96] Postgres bound to 36715
2026-09-27 22:03:46.412 UTC [15666] FATAL: database "slave" does not exist
127.3.159.20:36715 - accepting connections
I20260927 22:03:46.413300 3708 mini_ranger.cc:161] Starting Ranger...
I20260927 22:03:46.446557 3708 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 22:03:47.054960 3708 mini_ranger.cc:87] Created ranger Postgres database
I20260927 22:03:47.055181 3708 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-admin
2026-09-27 22:03:47,394 [I] DB FLAVOR :POSTGRES
2026-09-27 22:03:47,394 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:03:47,395 [I] Checking connection..
2026-09-27 22:03:47,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 22:03:47,701 [I] Checking connection passed.
2026-09-27 22:03:47,702 [I] --------- Verifying version history table ---------
2026-09-27 22:03:47,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:03:48,013 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:03:48,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:03:48,311 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 22:03:48,312 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 22:03:48,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 22:03:48.598 UTC [15730] WARNING: there is no transaction in progress
2026-09-27 22:03:48,607 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 22:03:48,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 22:03:48,901 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 22:03:48,901 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:03:48,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 22:03:49,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 22:03:49,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-sg4j.c.gcp-upstream.internal','N') ;"
2026-09-27 22:03:49,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:03:50,066 [I] Table x_portal_user does not exist in database ranger
2026-09-27 22:03:50,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:03:50,396 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 22:03:50,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:03:50,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:03:50,989 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 22:03:50,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 22:03:52.228 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.250 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.271 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.293 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.316 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.340 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.407 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.416 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.434 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.446 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.458 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.475 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.487 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.495 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.504 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:52.511 UTC [15838] WARNING: there is no transaction in progress
2026-09-27 22:03:53,446 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 22:03:53,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 22:03:53,717 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 22:03:53,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 22:03:54,017 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 22:03:54,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 22:03:54,330 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 22:03:54,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 22:03:54,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 22:03:54,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-sg4j.c.gcp-upstream.internal';"
2026-09-27 22:03:55,238 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 22:03:55,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 22:03:55,557 [I] DB_PATCHES have already been applied
I20260927 22:03:55.563763 3708 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIPxQZD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 22:03:55.563894 3708 mini_ranger.cc:193] Using host: 127.3.159.20
I20260927 22:03:55.582602 3708 mini_ranger.cc:227] Ranger admin URL: http://127.3.159.20:37151
Sep 27, 2026 10:03:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:03:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 10:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 10:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 10:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 10:03:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 22:03:56.800705 3708 mini_ranger.cc:160] Time spent starting Ranger: real 10.387s user 0.006s sys 0.049s
Sep 27, 2026 10:04:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 10:04:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 10:04:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 10:04:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 10:04:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 22:04:25.514813 3708 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.159.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.159.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/rangerkms_127.3.159.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.159.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/rangerkms_127.3.159.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.159.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.159.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.159.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/HTTP_127.3.159.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.159.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/HTTP_127.3.159.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 22:04:25 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546665, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 22:04:25.748656 3708 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 22:04:25.779215 3708 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 22:04:27.430033 3708 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 22:04:27.430212 3708 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-kms
I20260927 22:04:27.430262 3708 mini_ranger_kms.cc:227] Using postgres at 127.3.159.20:36715
2026-09-27 22:04:27,916 [I] DB FLAVOR :POSTGRES
2026-09-27 22:04:27,916 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 22:04:27,916 [I] Checking connection
2026-09-27 22:04:27,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 22:04:28,249 [I] connection success
2026-09-27 22:04:28,249 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 22:04:28,249 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 22:04:28,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 22:04:28,592 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 22:04:28,593 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 22:04:28,593 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 22:04:28,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.159.20:36715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 22:04:28,982 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 22:04:29.803278 3708 mini_ranger_kms.cc:310] Created kms service
I20260927 22:04:30.124027 3708 mini_ranger_kms.cc:326] Created kudu user
I20260927 22:04:30.205988 3708 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 22:04:30 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546665, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:04:30.819944 3708 mini_ranger_kms.cc:384] Added ranger policy
I20260927 22:04:30.820279 3708 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIPxQZD/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIPxQZD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 22:04:30.820324 3708 mini_ranger_kms.cc:240] Using host: 127.3.159.20
I20260927 22:04:30.842768 3708 mini_ranger_kms.cc:276] Ranger KMS PID: 16019
I20260927 22:04:30.843561 3708 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.159.20:46163
Sep 27, 2026 10:04:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 10:04:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 10:04:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 10:04:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 10:04:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 10:04:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 22:04:33.542922 3708 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.794s user 0.091s sys 0.425s
I20260927 22:04:33.543195 3708 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 22:04:33.543380 3708 mini_ranger_kms.cc:401] 127.3.159.20:46163/kms/v1/keys
Sep 27 22:04:33 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790546673, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.159.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 22:04:34 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546673, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.159.20@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
Sep 27 22:04:34 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546673, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.159.20@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
Sep 27 22:04:38 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:04:38.227497 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:40619
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:40619
--ranger_config_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:46163/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:04:38.689914 16069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:38.690361 16069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:38.690471 16069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:38.700518 16069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:04:38.700702 16069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:38.700762 16069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:38.700820 16069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:04:38.700870 16069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:04:38.714715 16069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.159.20:46163/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:40619
--ranger_config_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:40619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:38.716997 16069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:38.718945 16069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:38.736605 16074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:38.736470 16075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:38.736429 16077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:38.737866 16069 server_base.cc:1034] running on GCE node
I20260927 22:04:38.739002 16069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:38.741564 16069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:38.742995 16069 hybrid_clock.cc:648] HybridClock initialized: now 1790546678742906 us; error 92 us; skew 500 ppm
I20260927 22:04:38.743535 16069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:38 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:38.747891 16069 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:04:38.750276 16069 webserver.cc:460] Webserver started at http://127.3.159.62:45293/ using document root <none> and password file <none>
I20260927 22:04:38.751158 16069 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:38.751344 16069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:04:38.751935 16069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:04:38 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546678, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:04:38.843107 16069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "98a53bfca55c4f78bd8d2f43fe91e95f"
format_stamp: "Formatted at 2026-09-27 22:04:38 on dist-test-slave-sg4j"
server_key: "431b380218fff4bef8a332525a17ecab"
server_key_iv: "1bede3a574d8ae010d56af5f0306de01"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:38.844666 16069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "98a53bfca55c4f78bd8d2f43fe91e95f"
format_stamp: "Formatted at 2026-09-27 22:04:38 on dist-test-slave-sg4j"
server_key: "431b380218fff4bef8a332525a17ecab"
server_key_iv: "1bede3a574d8ae010d56af5f0306de01"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:38.854988 16069 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260927 22:04:38.888600 16084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:38.891194 16069 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:04:38.891495 16069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "98a53bfca55c4f78bd8d2f43fe91e95f"
format_stamp: "Formatted at 2026-09-27 22:04:38 on dist-test-slave-sg4j"
server_key: "431b380218fff4bef8a332525a17ecab"
server_key_iv: "1bede3a574d8ae010d56af5f0306de01"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:38.891783 16069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:04:38.925694 16069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:38.932444 16069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:38.932868 16069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:38.964989 16069 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:40619
I20260927 22:04:38.965297 16135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:40619 every 8 connection(s)
I20260927 22:04:38.969287 16069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:04:38.969864 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 16069
I20260927 22:04:38.970348 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:04:38.973019 3708 external_mini_cluster.cc:1428] Setting key 6931122832d5de94d2891878703dc681
I20260927 22:04:38.978873 16136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:04:39.005709 16136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f: Bootstrap starting.
Sep 27 22:04:39 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:04:39.011046 16136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f: Neither blocks nor log segments found. Creating new log.
I20260927 22:04:39.014948 16136 log.cc:824] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:04:39.020478 16136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f: No bootstrap required, opened a new log
I20260927 22:04:39.028126 16139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:38.983379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.3.159.62:40619)
0927 22:04:38.984524 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:38.984548 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:38.984626 (+ 78us) server_negotiation.cc:407] Connection header received
0927 22:04:38.990197 (+ 5571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:38.990265 (+ 68us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:38.991291 (+ 1026us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:38.992119 (+ 828us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:38.994904 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:38.997656 (+ 2752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:38.999757 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:39.001916 (+ 2159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:39.002298 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:39.009539 (+ 7241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:39.009553 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:39.009579 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:39.009671 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:39.019428 (+ 9757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:39.021646 (+ 2218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:39.021659 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:39.021671 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:39.021969 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:39.022525 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:39.022529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:39.022532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:39.023885 (+ 1353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:39.024394 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:39.025276 (+ 882us) server_negotiation.cc:299] Negotiation successful
0927 22:04:39.026273 (+ 997us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":466,"threads_started":1}
I20260927 22:04:39.034194 16136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } }
I20260927 22:04:39.034725 16136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:04:39.034844 16136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a53bfca55c4f78bd8d2f43fe91e95f, State: Initialized, Role: FOLLOWER
I20260927 22:04:39.035547 16136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } }
I20260927 22:04:39.035794 16136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:04:39.035953 16136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:04:39.036167 16136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:04:39.039021 16136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } }
I20260927 22:04:39.040056 16136 leader_election.cc:302] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98a53bfca55c4f78bd8d2f43fe91e95f; no voters:
I20260927 22:04:39.040964 16136 leader_election.cc:288] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:04:39.041980 16141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:04:39.044510 16141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 1 LEADER]: Becoming Leader. State: Replica: 98a53bfca55c4f78bd8d2f43fe91e95f, State: Running, Role: LEADER
I20260927 22:04:39.045678 16136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:04:39.045615 16141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } }
I20260927 22:04:39.050454 16142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } } }
I20260927 22:04:39.051090 16142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [sys.catalog]: This master's current role is: LEADER
I20260927 22:04:39.050578 16143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98a53bfca55c4f78bd8d2f43fe91e95f. Latest consensus state: current_term: 1 leader_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } } }
I20260927 22:04:39.051261 16143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [sys.catalog]: This master's current role is: LEADER
I20260927 22:04:39.057974 16149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:04:39.068770 16149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:04:39.072383 16136 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 22:04:39.101840 16149 catalog_manager.cc:1347] Generated new cluster ID: 00dc994024ef4257bf4b588c000be22a
I20260927 22:04:39.102121 16149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:04:39.148648 16149 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:04:39.151548 16149 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:04:39.185606 16149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f: Generated new TSK 0
I20260927 22:04:39.187654 16149 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
W20260927 22:04:40.416729 3708 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.159.1: real 0.108s user 0.003s sys 0.031s
I20260927 22:04:40.417989 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:46163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40619
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:04:40 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790546680, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:04:41.044486 16181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:41.045075 16181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:41.045347 16181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:41.059326 16181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:41.059607 16181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:41.059854 16181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:04:41.078389 16181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.159.20:46163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:41.080566 16181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:41.082682 16181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:41.103130 16192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:41.103505 16194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:41.104427 16191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:41.105553 16181 server_base.cc:1034] running on GCE node
I20260927 22:04:41.106386 16181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:41.107736 16181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:41.109045 16181 hybrid_clock.cc:648] HybridClock initialized: now 1790546681108932 us; error 112 us; skew 500 ppm
I20260927 22:04:41.109550 16181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:41 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546681, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:41.115634 16181 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:04:41.138527 16181 webserver.cc:460] Webserver started at http://127.3.159.1:37775/ using document root <none> and password file <none>
I20260927 22:04:41.139914 16181 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:41.140136 16181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:04:41.140718 16181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:04:41 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546681, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:04:41.168644 16181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "8e3d2b0688114625bfd477d42fc7f334"
format_stamp: "Formatted at 2026-09-27 22:04:41 on dist-test-slave-sg4j"
server_key: "6d3c3ec156277099675279f18e3d7621"
server_key_iv: "e97adc5f9ba7c41985620c89fb23ad97"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:41.170630 16181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "8e3d2b0688114625bfd477d42fc7f334"
format_stamp: "Formatted at 2026-09-27 22:04:41 on dist-test-slave-sg4j"
server_key: "6d3c3ec156277099675279f18e3d7621"
server_key_iv: "e97adc5f9ba7c41985620c89fb23ad97"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:41.194660 16181 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.019s sys 0.000s
I20260927 22:04:41.227628 16201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:41.233263 16181 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.000s
I20260927 22:04:41.234184 16181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "8e3d2b0688114625bfd477d42fc7f334"
format_stamp: "Formatted at 2026-09-27 22:04:41 on dist-test-slave-sg4j"
server_key: "6d3c3ec156277099675279f18e3d7621"
server_key_iv: "e97adc5f9ba7c41985620c89fb23ad97"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:41.235956 16181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:04:41.317245 16181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:41.324126 16181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:41.324590 16181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:41.332407 16181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:04:41.341127 16181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:04:41.341362 16181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:41.341632 16181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:04:41.341802 16181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 22:04:41 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546680, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
Sep 27 22:04:42 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546680, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:04:42.038494 16181 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:35163
I20260927 22:04:42.038451 16314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:35163 every 8 connection(s)
I20260927 22:04:42.041450 16181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:04:42.045838 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 16181
I20260927 22:04:42.046140 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:04:42.047066 3708 external_mini_cluster.cc:1428] Setting key 471614eb7c0d5ab34d7853dba4175c0b
Sep 27 22:04:42 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546681, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:04:42.102416 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:42.048655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:34389 (local address 127.3.159.62:40619)
0927 22:04:42.051136 (+ 2481us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:42.051157 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:42.074353 (+ 23196us) server_negotiation.cc:407] Connection header received
0927 22:04:42.074453 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:42.074459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:42.074595 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:42.074751 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:42.077363 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:42.078670 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:42.081614 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:42.082421 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:42.082552 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:42.089789 (+ 7237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:42.089799 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:42.089803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:42.089861 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:42.096348 (+ 6487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:42.097937 (+ 1589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:42.097944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:42.097948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:42.098160 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:42.098910 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:42.098916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:42.098919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:42.099357 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:42.099531 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:42.101625 (+ 2094us) server_negotiation.cc:299] Negotiation successful
0927 22:04:42.101947 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2300,"thread_start_us":2069,"threads_started":1}
I20260927 22:04:42.105340 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:42.047448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:40619 (local address 127.3.159.1:34389)
0927 22:04:42.070153 (+ 22705us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:42.070244 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:42.073770 (+ 3526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:42.075304 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:42.075318 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:42.076113 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:42.077066 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:42.077095 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:42.078946 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:42.078953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:42.081315 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:42.081339 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:42.083067 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:42.083077 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:42.083446 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:42.084709 (+ 1263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:42.084747 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:42.089290 (+ 4543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:42.096688 (+ 7398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:42.096702 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:42.096726 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:42.097699 (+ 973us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:42.098359 (+ 660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:42.098366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:42.098370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:42.098707 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:42.099622 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:42.099631 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:42.100180 (+ 549us) client_negotiation.cc:769] Sending connection context
0927 22:04:42.100587 (+ 407us) client_negotiation.cc:240] Negotiation successful
0927 22:04:42.101268 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22238,"thread_start_us":21867,"threads_started":1}
I20260927 22:04:42.108140 16315 heartbeater.cc:347] Connected to a master server at 127.3.159.62:40619
I20260927 22:04:42.108587 16315 heartbeater.cc:464] Registering TS with master...
I20260927 22:04:42.110093 16315 heartbeater.cc:511] Master 127.3.159.62:40619 requested a full tablet report, sending...
I20260927 22:04:42.114471 16101 ts_manager.cc:194] Registered new tserver with Master: 8e3d2b0688114625bfd477d42fc7f334 (127.3.159.1:35163)
I20260927 22:04:42.117218 16101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:34389
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
W20260927 22:04:42.197469 3708 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.159.2: real 0.145s user 0.003s sys 0.019s
I20260927 22:04:42.198902 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:46163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40619
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:04:42.917058 16322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:42.917646 16322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:42.917876 16322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:42.927088 16322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:42.927305 16322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:42.927487 16322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:04:42.938889 16322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.159.20:46163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:42.940460 16322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:42.942154 16322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:42.962890 16337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:42.963574 16335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:42.964318 16334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:42.965924 16322 server_base.cc:1034] running on GCE node
I20260927 22:04:42.966868 16322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:42.968288 16322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:42.969830 16322 hybrid_clock.cc:648] HybridClock initialized: now 1790546682969755 us; error 79 us; skew 500 ppm
I20260927 22:04:42.970364 16322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:42 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546682, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:42.981266 16322 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:04:42.983889 16322 webserver.cc:460] Webserver started at http://127.3.159.2:36513/ using document root <none> and password file <none>
I20260927 22:04:42.984845 16322 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:42.985028 16322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:04:42.985498 16322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:04:42 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546682, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:04:43.008992 16322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "2bf311513186424aa56c8a1a812f73d4"
format_stamp: "Formatted at 2026-09-27 22:04:42 on dist-test-slave-sg4j"
server_key: "58c960c92bf34cceb3373091905a3215"
server_key_iv: "4594de63866f02294d71066c9d62dd7f"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:43.010264 16322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "2bf311513186424aa56c8a1a812f73d4"
format_stamp: "Formatted at 2026-09-27 22:04:42 on dist-test-slave-sg4j"
server_key: "58c960c92bf34cceb3373091905a3215"
server_key_iv: "4594de63866f02294d71066c9d62dd7f"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:43.018505 16322 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260927 22:04:43.042282 16344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:43.045369 16322 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.003s
I20260927 22:04:43.045687 16322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "2bf311513186424aa56c8a1a812f73d4"
format_stamp: "Formatted at 2026-09-27 22:04:42 on dist-test-slave-sg4j"
server_key: "58c960c92bf34cceb3373091905a3215"
server_key_iv: "4594de63866f02294d71066c9d62dd7f"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:43.046003 16322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:04:43.079928 16322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:43.087937 16322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:43.088361 16322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:43.090025 16322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:04:43.092259 16322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:04:43.092381 16322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:43.092736 16322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:04:43.092859 16322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:43.122438 16315 heartbeater.cc:503] Master 127.3.159.62:40619 was elected leader, sending a full tablet report...
I20260927 22:04:43.159092 16322 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:33665
I20260927 22:04:43.159291 16456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:33665 every 8 connection(s)
I20260927 22:04:43.160693 16322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:04:43.162966 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 16322
I20260927 22:04:43.163282 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:04:43.163791 3708 external_mini_cluster.cc:1428] Setting key 72e34ae301d966e4991d1abbba70183f
Sep 27 22:04:43 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546682, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
I20260927 22:04:43.203367 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:43.165350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:56073 (local address 127.3.159.62:40619)
0927 22:04:43.166263 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:43.166272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:43.168038 (+ 1766us) server_negotiation.cc:407] Connection header received
0927 22:04:43.172864 (+ 4826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:43.172873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:43.173024 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:43.173187 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:43.176407 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:43.178087 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:43.180610 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:43.181456 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:43.181671 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:43.188763 (+ 7092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:43.188775 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:43.188784 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:43.188857 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:43.194196 (+ 5339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:43.199039 (+ 4843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:43.199051 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:43.199057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:43.199307 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:43.200241 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:43.200246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:43.200249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:43.200606 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:43.200742 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:43.202416 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0927 22:04:43.202800 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":393,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
I20260927 22:04:43.205181 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:43.166134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:40619 (local address 127.3.159.2:56073)
0927 22:04:43.167172 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:43.167843 (+ 671us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:43.171722 (+ 3879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:43.173823 (+ 2101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:43.173843 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:43.174741 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:43.175885 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:43.175925 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:43.178444 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:43.178454 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:43.180329 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:43.180352 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:43.181921 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:43.181937 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:43.182565 (+ 628us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:43.184003 (+ 1438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:43.184045 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:43.187945 (+ 3900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:43.196409 (+ 8464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:43.196426 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:43.196457 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:43.198693 (+ 2236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:43.199559 (+ 866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:43.199565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:43.199569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:43.199977 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:43.200816 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:43.200828 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:43.201643 (+ 815us) client_negotiation.cc:769] Sending connection context
0927 22:04:43.202093 (+ 450us) client_negotiation.cc:240] Negotiation successful
0927 22:04:43.202907 (+ 814us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":444,"threads_started":1}
I20260927 22:04:43.208014 16457 heartbeater.cc:347] Connected to a master server at 127.3.159.62:40619
I20260927 22:04:43.208539 16457 heartbeater.cc:464] Registering TS with master...
I20260927 22:04:43.210078 16457 heartbeater.cc:511] Master 127.3.159.62:40619 requested a full tablet report, sending...
I20260927 22:04:43.213328 16101 ts_manager.cc:194] Registered new tserver with Master: 2bf311513186424aa56c8a1a812f73d4 (127.3.159.2:33665)
I20260927 22:04:43.215001 16101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:56073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:04:43.243595 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:46163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40619
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:04:43.692723 16463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:43.693147 16463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:43.693318 16463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:43.701994 16463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:43.702142 16463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:43.702340 16463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:04:43.714118 16463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.159.20:46163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:43.715893 16463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:43.717670 16463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:43.735368 16469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:43.735553 16468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:43.735532 16471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:43.736447 16463 server_base.cc:1034] running on GCE node
I20260927 22:04:43.738055 16463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:43.739495 16463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:43.740892 16463 hybrid_clock.cc:648] HybridClock initialized: now 1790546683740825 us; error 71 us; skew 500 ppm
I20260927 22:04:43.741693 16463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:43 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546683, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:43.746634 16463 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:04:43.749223 16463 webserver.cc:460] Webserver started at http://127.3.159.3:42005/ using document root <none> and password file <none>
I20260927 22:04:43.750411 16463 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:43.750622 16463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:04:43.751055 16463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 22:04:43 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546683, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:04:43.771696 16463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "df8da39e95ed4f53aeacc85a5c452fbe"
format_stamp: "Formatted at 2026-09-27 22:04:43 on dist-test-slave-sg4j"
server_key: "0a0724b711325913b12c3411eafe89d0"
server_key_iv: "1cc38c1048b0063b4456d644dfca0564"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:43.773275 16463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "df8da39e95ed4f53aeacc85a5c452fbe"
format_stamp: "Formatted at 2026-09-27 22:04:43 on dist-test-slave-sg4j"
server_key: "0a0724b711325913b12c3411eafe89d0"
server_key_iv: "1cc38c1048b0063b4456d644dfca0564"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:43.784093 16463 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.008s sys 0.000s
I20260927 22:04:43.805704 16478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:43.809173 16463 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260927 22:04:43.809531 16463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "df8da39e95ed4f53aeacc85a5c452fbe"
format_stamp: "Formatted at 2026-09-27 22:04:43 on dist-test-slave-sg4j"
server_key: "0a0724b711325913b12c3411eafe89d0"
server_key_iv: "1cc38c1048b0063b4456d644dfca0564"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:43.809922 16463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:04:43.827337 16463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:43.833746 16463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:43.834151 16463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:43.835482 16463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:04:43.837841 16463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:04:43.837951 16463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:43.838073 16463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:04:43.838119 16463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:43.896008 16463 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:34843
I20260927 22:04:43.896046 16590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:34843 every 8 connection(s)
I20260927 22:04:43.897923 16463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:04:43.901607 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 16463
I20260927 22:04:43.901820 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:04:43.902488 3708 external_mini_cluster.cc:1428] Setting key 202d0e9d3b1873399b061e3bc0d4a3fa
Sep 27 22:04:43 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546683, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:04:43.946730 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:43.901811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:38075 (local address 127.3.159.62:40619)
0927 22:04:43.902525 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:43.902533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:43.915043 (+ 12510us) server_negotiation.cc:407] Connection header received
0927 22:04:43.917938 (+ 2895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:43.917946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:43.918057 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:43.918490 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:43.921454 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:43.923354 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:43.927179 (+ 3825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:43.927963 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:43.928088 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:43.934739 (+ 6651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:43.934749 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:43.934755 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:43.934842 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:43.939231 (+ 4389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:43.941317 (+ 2086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:43.941327 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:43.941331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:43.941658 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:43.942410 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:43.942419 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:43.942423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:43.943083 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:43.943407 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:43.945854 (+ 2447us) server_negotiation.cc:299] Negotiation successful
0927 22:04:43.946325 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":302,"threads_started":1}
I20260927 22:04:43.947594 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:43.913349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:40619 (local address 127.3.159.3:38075)
0927 22:04:43.914800 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:43.914872 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:43.917440 (+ 2568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:43.918879 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:43.918897 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:43.919822 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:43.921047 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:43.921092 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:43.923672 (+ 2580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:43.923680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:43.926726 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:43.926770 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:43.928630 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:43.928642 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:43.929059 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:43.930190 (+ 1131us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:43.930237 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:43.934209 (+ 3972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:43.939516 (+ 5307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:43.939536 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:43.939567 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:43.940498 (+ 931us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:43.941882 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:43.941889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:43.941893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:43.942199 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:43.943358 (+ 1159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:43.943372 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:43.943950 (+ 578us) client_negotiation.cc:769] Sending connection context
0927 22:04:43.944375 (+ 425us) client_negotiation.cc:240] Negotiation successful
0927 22:04:43.945029 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":952,"thread_start_us":610,"threads_started":1}
I20260927 22:04:43.949990 16591 heartbeater.cc:347] Connected to a master server at 127.3.159.62:40619
I20260927 22:04:43.950345 16591 heartbeater.cc:464] Registering TS with master...
I20260927 22:04:43.951162 16591 heartbeater.cc:511] Master 127.3.159.62:40619 requested a full tablet report, sending...
I20260927 22:04:43.953308 16101 ts_manager.cc:194] Registered new tserver with Master: df8da39e95ed4f53aeacc85a5c452fbe (127.3.159.3:34843)
I20260927 22:04:43.954466 16101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:38075
I20260927 22:04:43.958796 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:04:43.960793 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 16181
Sep 27 22:04:43 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:04:43.987412 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:43.962825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.3.159.1:35163)
0927 22:04:43.963990 (+ 1165us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:43.963998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:43.964026 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:04:43.964095 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:43.964101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:43.964314 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:43.964467 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:43.967036 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:43.968220 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:43.970746 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:43.971503 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:43.971676 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:43.978809 (+ 7133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:43.978836 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:43.978861 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:43.978930 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:43.982752 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:43.984104 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:43.984115 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:43.984118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:43.984333 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:43.984926 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:43.984931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:43.984933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:43.985530 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:43.985875 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:43.986505 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 22:04:43.986926 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":791,"threads_started":1}
I20260927 22:04:43.989243 16310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:44.085011 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 16181
I20260927 22:04:44.105798 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 16322
Sep 27 22:04:44 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:04:44.127434 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:44.106978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.3.159.2:33665)
0927 22:04:44.107682 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:44.107691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:44.107721 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:04:44.107809 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:44.107818 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:44.108136 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:44.108354 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:44.110088 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:44.111221 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:44.112745 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:44.113793 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:44.113960 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:44.119661 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:44.119681 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:44.119707 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:44.119758 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:44.123783 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:44.124789 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:44.124799 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:44.124802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:44.124982 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:44.125452 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:44.125458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:44.125462 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:44.125899 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:44.126284 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:44.126670 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 22:04:44.127001 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":385,"threads_started":1}
I20260927 22:04:44.129333 16452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:44.226737 16457 heartbeater.cc:503] Master 127.3.159.62:40619 was elected leader, sending a full tablet report...
I20260927 22:04:44.229031 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 16322
I20260927 22:04:44.248970 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 16463
Sep 27 22:04:44 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:04:44.273789 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:44.250634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57180 (local address 127.3.159.3:34843)
0927 22:04:44.251483 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:44.251491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:44.251531 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:04:44.251605 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:44.251613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:44.251864 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:44.252039 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:44.254070 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:44.255470 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:44.257222 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:44.258461 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:44.258630 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:44.264942 (+ 6312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:44.264974 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:44.264999 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:44.265061 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:44.269162 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:44.270444 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:44.270455 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:44.270458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:44.270675 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:44.271278 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:44.271283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:44.271286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:44.271814 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:44.272576 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:44.272772 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 22:04:44.273318 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":393,"threads_started":1}
I20260927 22:04:44.275688 16586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:44.378140 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 16463
I20260927 22:04:44.398938 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 16069
I20260927 22:04:44.401068 16131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:44.531553 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 16069
I20260927 22:04:44.559477 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:40619
--webserver_interface=127.3.159.62
--webserver_port=45293
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:40619
--ranger_config_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:46163/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.159.20:46163/kms with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:04:44.980919 16603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:44.981266 16603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:44.981410 16603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:44.989168 16603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:04:44.989313 16603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:44.989364 16603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:44.989516 16603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:04:44.989593 16603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:04:45.000993 16603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.159.20:46163/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:40619
--ranger_config_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:40619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=45293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:45.002724 16603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:45.004598 16603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:45.019320 16608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:45.019397 16611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:45.019620 16609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:45.021107 16603 server_base.cc:1034] running on GCE node
I20260927 22:04:45.022462 16603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:45.024765 16603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:45.026104 16603 hybrid_clock.cc:648] HybridClock initialized: now 1790546685026050 us; error 65 us; skew 500 ppm
I20260927 22:04:45.026634 16603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:45 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546685, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:45.031005 16603 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:04:45.033155 16603 webserver.cc:460] Webserver started at http://127.3.159.62:45293/ using document root <none> and password file <none>
I20260927 22:04:45.034121 16603 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:45.034312 16603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:04:45 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546685, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:04:45.093011 16603 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 22:04:45.098353 16619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:45.100493 16603 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:04:45.100797 16603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "98a53bfca55c4f78bd8d2f43fe91e95f"
format_stamp: "Formatted at 2026-09-27 22:04:38 on dist-test-slave-sg4j"
server_key: "431b380218fff4bef8a332525a17ecab"
server_key_iv: "1bede3a574d8ae010d56af5f0306de01"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:45.102101 16603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:04:45.125128 16603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:45.130251 16603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:45.130630 16603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:45.155920 16603 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:40619
I20260927 22:04:45.155956 16670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:40619 every 8 connection(s)
I20260927 22:04:45.158118 16603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:04:45.164717 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 16603
I20260927 22:04:45.165969 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:46163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:35163
--local_ip_for_outbound_sockets=127.3.159.1
--tserver_master_addrs=127.3.159.62:40619
--webserver_port=37775
--webserver_interface=127.3.159.1
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.159.20:46163/kms with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:04:45.167270 16671 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:04:45.171034 16671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f: Bootstrap starting.
I20260927 22:04:45.193312 16671 log.cc:824] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:04:45.205833 16671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:04:45.206588 16671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f: Bootstrap complete.
I20260927 22:04:45.217053 16671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } }
I20260927 22:04:45.218269 16671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a53bfca55c4f78bd8d2f43fe91e95f, State: Initialized, Role: FOLLOWER
I20260927 22:04:45.219673 16671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } }
I20260927 22:04:45.220270 16671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:04:45.220603 16671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:04:45.220876 16671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:04:45.225600 16671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } }
I20260927 22:04:45.226260 16671 leader_election.cc:302] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98a53bfca55c4f78bd8d2f43fe91e95f; no voters:
I20260927 22:04:45.226966 16671 leader_election.cc:288] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:04:45.227111 16675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:04:45.228744 16675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [term 2 LEADER]: Becoming Leader. State: Replica: 98a53bfca55c4f78bd8d2f43fe91e95f, State: Running, Role: LEADER
I20260927 22:04:45.229423 16675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } }
I20260927 22:04:45.229782 16671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:04:45.232434 16676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } } }
I20260927 22:04:45.233012 16676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [sys.catalog]: This master's current role is: LEADER
I20260927 22:04:45.233726 16677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98a53bfca55c4f78bd8d2f43fe91e95f. Latest consensus state: current_term: 2 leader_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a53bfca55c4f78bd8d2f43fe91e95f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 40619 } } }
I20260927 22:04:45.234503 16677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f [sys.catalog]: This master's current role is: LEADER
I20260927 22:04:45.240052 16682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:04:45.254781 16682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:04:45.256767 16682 catalog_manager.cc:1259] Loaded cluster ID: 00dc994024ef4257bf4b588c000be22a
I20260927 22:04:45.256906 16682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:04:45.261705 16682 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:04:45.265053 16682 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f: Loaded TSK: 0
I20260927 22:04:45.267820 16682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:04:45.271166 16671 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 22:04:45.754637 16673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:45.755308 16673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:45.755740 16673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:45.771499 16673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:45.771744 16673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:45.772178 16673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:04:45.794801 16673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.159.20:46163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:35163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=37775
--tserver_master_addrs=127.3.159.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:45.797336 16673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:45.800339 16673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:45.838896 16709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:45.839438 16710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:45.845918 16673 server_base.cc:1034] running on GCE node
W20260927 22:04:45.846206 16712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:45.847469 16673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:45.871698 16673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:45.873610 16673 hybrid_clock.cc:648] HybridClock initialized: now 1790546685873636 us; error 225 us; skew 500 ppm
I20260927 22:04:45.874337 16673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:45 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546685, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:45.883286 16673 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:04:45.898413 16673 webserver.cc:460] Webserver started at http://127.3.159.1:37775/ using document root <none> and password file <none>
I20260927 22:04:45.899709 16673 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:45.899924 16673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:04:45 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546685, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:04:45.938395 16673 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.001s
I20260927 22:04:45.947355 16720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:45.950992 16673 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.005s
I20260927 22:04:45.951334 16673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "8e3d2b0688114625bfd477d42fc7f334"
format_stamp: "Formatted at 2026-09-27 22:04:41 on dist-test-slave-sg4j"
server_key: "6d3c3ec156277099675279f18e3d7621"
server_key_iv: "e97adc5f9ba7c41985620c89fb23ad97"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:45.953359 16673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:04:46.011819 16673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:46.018769 16673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:46.019191 16673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:46.020874 16673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:04:46.024426 16673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:04:46.024587 16673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:46.024732 16673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:04:46.024911 16673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:46.120813 16673 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:35163
I20260927 22:04:46.120906 16832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:35163 every 8 connection(s)
I20260927 22:04:46.123827 16673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:04:46.132630 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 16673
I20260927 22:04:46.134003 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:46163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:33665
--local_ip_for_outbound_sockets=127.3.159.2
--tserver_master_addrs=127.3.159.62:40619
--webserver_port=36513
--webserver_interface=127.3.159.2
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.159.20:46163/kms with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:04:46 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546685, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:04:46.176692 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:46.129162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:40619 (local address 127.3.159.1:53989)
0927 22:04:46.144460 (+ 15298us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:46.144541 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:46.147126 (+ 2585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:46.149668 (+ 2542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:46.149682 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:46.150538 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:46.151586 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:46.151627 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:46.154362 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:46.154368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:46.156707 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:46.156735 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:46.158969 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:46.158981 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:46.159310 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:46.160325 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:46.160367 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:46.163870 (+ 3503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:46.169622 (+ 5752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:46.169635 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:46.169657 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:46.170508 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:46.171155 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:46.171160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:46.171164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:46.171474 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:46.172972 (+ 1498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:46.172984 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:46.173601 (+ 617us) client_negotiation.cc:769] Sending connection context
0927 22:04:46.174022 (+ 421us) client_negotiation.cc:240] Negotiation successful
0927 22:04:46.174871 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14925,"thread_start_us":14607,"threads_started":1}
I20260927 22:04:46.178223 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:46.129137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:53989 (local address 127.3.159.62:40619)
0927 22:04:46.143498 (+ 14361us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:46.143537 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:46.144766 (+ 1229us) server_negotiation.cc:407] Connection header received
0927 22:04:46.148124 (+ 3358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:46.148142 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:46.148730 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:46.149262 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:46.152014 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:46.154077 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:46.156970 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:46.158224 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:46.158452 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:46.164361 (+ 5909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:46.164379 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:46.164405 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:46.164480 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:46.169292 (+ 4812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:46.170732 (+ 1440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:46.170741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:46.170749 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:46.170999 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:46.171634 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:46.171639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:46.171643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:46.172600 (+ 957us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:46.172857 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:46.175550 (+ 2693us) server_negotiation.cc:299] Negotiation successful
0927 22:04:46.176161 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13978,"thread_start_us":13666,"threads_started":1}
I20260927 22:04:46.184067 16833 heartbeater.cc:347] Connected to a master server at 127.3.159.62:40619
I20260927 22:04:46.184613 16833 heartbeater.cc:464] Registering TS with master...
I20260927 22:04:46.186048 16833 heartbeater.cc:511] Master 127.3.159.62:40619 requested a full tablet report, sending...
W20260927 22:04:46.192030 16833 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:40619 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:04:46.192669 16833 heartbeater.cc:464] Registering TS with master...
I20260927 22:04:46.193253 16833 heartbeater.cc:511] Master 127.3.159.62:40619 requested a full tablet report, sending...
I20260927 22:04:46.195677 16833 heartbeater.cc:464] Registering TS with master...
I20260927 22:04:46.196290 16833 heartbeater.cc:511] Master 127.3.159.62:40619 requested a full tablet report, sending...
W20260927 22:04:46.198253 16833 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 22:04:46 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790546686, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 22:04:46.681515 16837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:46.682168 16837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:46.682384 16837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:46.697723 16837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:46.697999 16837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:46.698237 16837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:04:46.719296 16837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.159.20:46163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:33665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=36513
--tserver_master_addrs=127.3.159.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:46.721905 16837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:46.724535 16837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:46.749619 16853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:46.761461 16854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:46.766824 16856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:46.767727 16837 server_base.cc:1034] running on GCE node
I20260927 22:04:46.769193 16837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:46.770968 16837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:46.774510 16837 hybrid_clock.cc:648] HybridClock initialized: now 1790546686774387 us; error 89 us; skew 500 ppm
I20260927 22:04:46.775223 16837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:46 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546686, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:46.781044 16837 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:04:46.784186 16837 webserver.cc:460] Webserver started at http://127.3.159.2:36513/ using document root <none> and password file <none>
I20260927 22:04:46.785585 16837 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:46.785806 16837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:04:46 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546686, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:04:46.824005 16837 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.004s sys 0.000s
I20260927 22:04:46.829346 16865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:46.832640 16837 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260927 22:04:46.833039 16837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "2bf311513186424aa56c8a1a812f73d4"
format_stamp: "Formatted at 2026-09-27 22:04:42 on dist-test-slave-sg4j"
server_key: "58c960c92bf34cceb3373091905a3215"
server_key_iv: "4594de63866f02294d71066c9d62dd7f"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:46.834950 16837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:04:46.867203 16837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:46.873781 16837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:46.874125 16837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:46.876472 16837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:04:46.878971 16837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:04:46.879182 16837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:46.879326 16837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:04:46.879415 16837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:46.979681 16837 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:33665
I20260927 22:04:46.979720 16977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:33665 every 8 connection(s)
I20260927 22:04:46.982053 16837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:04:46.986956 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 16837
I20260927 22:04:46.987841 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.159.20:46163/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:34843
--local_ip_for_outbound_sockets=127.3.159.3
--tserver_master_addrs=127.3.159.62:40619
--webserver_port=42005
--webserver_interface=127.3.159.3
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.159.20:46163/kms with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:04:47 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546686, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:04:47.026531 16980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:46.987382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:58095 (local address 127.3.159.62:40619)
0927 22:04:46.991779 (+ 4397us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:46.991793 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:46.996195 (+ 4402us) server_negotiation.cc:407] Connection header received
0927 22:04:47.000015 (+ 3820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:47.000025 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:47.000166 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:47.000272 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:47.003368 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:47.004430 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:47.007779 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:47.008531 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:47.008680 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:47.015507 (+ 6827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:47.015515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:47.015519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:47.015573 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:47.019922 (+ 4349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:47.021654 (+ 1732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:47.021663 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:47.021667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:47.021921 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:47.022630 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:47.022643 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:47.022646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:47.023179 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:47.023827 (+ 648us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:47.025461 (+ 1634us) server_negotiation.cc:299] Negotiation successful
0927 22:04:47.026000 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4117,"thread_start_us":3894,"threads_started":1}
I20260927 22:04:47.030124 16981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:46.988551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:40619 (local address 127.3.159.2:58095)
0927 22:04:46.995867 (+ 7316us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:46.995961 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:46.999286 (+ 3325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:47.001188 (+ 1902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:47.001206 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:47.002153 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:47.003060 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:47.003087 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:47.004650 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:47.004655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:47.007394 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:47.007425 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:47.009270 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:47.009280 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:47.009740 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:47.011049 (+ 1309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:47.011121 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:47.015052 (+ 3931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:47.020425 (+ 5373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:47.020440 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:47.020469 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:47.021416 (+ 947us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:47.022142 (+ 726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:47.022146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:47.022149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:47.022455 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:47.023953 (+ 1498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:47.023979 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:47.024722 (+ 743us) client_negotiation.cc:769] Sending connection context
0927 22:04:47.027035 (+ 2313us) client_negotiation.cc:240] Negotiation successful
0927 22:04:47.028010 (+ 975us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6855,"thread_start_us":6463,"threads_started":1}
I20260927 22:04:47.033284 16978 heartbeater.cc:347] Connected to a master server at 127.3.159.62:40619
I20260927 22:04:47.033856 16978 heartbeater.cc:464] Registering TS with master...
I20260927 22:04:47.035127 16978 heartbeater.cc:511] Master 127.3.159.62:40619 requested a full tablet report, sending...
I20260927 22:04:47.038446 16636 ts_manager.cc:194] Registered new tserver with Master: 2bf311513186424aa56c8a1a812f73d4 (127.3.159.2:33665)
I20260927 22:04:47.041443 16636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:58095
I20260927 22:04:47.198410 16833 heartbeater.cc:464] Registering TS with master...
I20260927 22:04:47.198957 16833 heartbeater.cc:511] Master 127.3.159.62:40619 requested a full tablet report, sending...
I20260927 22:04:47.200440 16636 ts_manager.cc:194] Registered new tserver with Master: 8e3d2b0688114625bfd477d42fc7f334 (127.3.159.1:35163)
I20260927 22:04:47.201932 16636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:53989
Sep 27 22:04:47 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546686, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
W20260927 22:04:47.570428 16982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:47.571034 16982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:47.571239 16982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:47.586103 16982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:47.586345 16982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:47.586593 16982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:04:47.607220 16982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.159.20:46163/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:34843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=42005
--tserver_master_addrs=127.3.159.62:40619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:47.609707 16982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:47.612224 16982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:47.637284 16987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:47.637494 16988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:47.655403 16990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:47.655879 16982 server_base.cc:1034] running on GCE node
I20260927 22:04:47.656783 16982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:47.660147 16982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:47.662387 16982 hybrid_clock.cc:648] HybridClock initialized: now 1790546687662239 us; error 107 us; skew 500 ppm
I20260927 22:04:47.662945 16982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:47 dist-test-slave-sg4j krb5kdc[15625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546687, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:47.668892 16982 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:04:47.678122 16982 webserver.cc:460] Webserver started at http://127.3.159.3:42005/ using document root <none> and password file <none>
I20260927 22:04:47.679355 16982 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:47.679586 16982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 22:04:47 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546687, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:04:47.725665 16982 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.010s sys 0.000s
I20260927 22:04:47.730201 16999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:47.732528 16982 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:04:47.733134 16982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "df8da39e95ed4f53aeacc85a5c452fbe"
format_stamp: "Formatted at 2026-09-27 22:04:43 on dist-test-slave-sg4j"
server_key: "0a0724b711325913b12c3411eafe89d0"
server_key_iv: "1cc38c1048b0063b4456d644dfca0564"
server_key_version: "kuduclusterkey@0"
I20260927 22:04:47.735033 16982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 22:04:47 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546686, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for HTTP/127.3.159.20@KRBTEST.COM
I20260927 22:04:47.780717 16982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:47.789780 16982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:47.790163 16982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:47.798099 16982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:04:47.810767 16982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:04:47.810931 16982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:47.811082 16982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:04:47.811182 16982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:48.049837 16978 heartbeater.cc:503] Master 127.3.159.62:40619 was elected leader, sending a full tablet report...
I20260927 22:04:48.205873 16833 heartbeater.cc:503] Master 127.3.159.62:40619 was elected leader, sending a full tablet report...
I20260927 22:04:48.254408 16982 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:34843
I20260927 22:04:48.254529 17118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:34843 every 8 connection(s)
I20260927 22:04:48.256764 16982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:04:48.264541 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 16982
Sep 27 22:04:48 dist-test-slave-sg4j krb5kdc[15625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546687, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:04:48.303932 17124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.267532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50830 (local address 127.3.159.62:40619)
0927 22:04:48.273066 (+ 5534us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:48.273073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:48.273094 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:04:48.273153 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:48.273158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:48.273244 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:48.273369 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:48.276140 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.277582 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.280436 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.281184 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.281364 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.292206 (+ 10842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:48.292216 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:48.292221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:48.292306 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:48.298662 (+ 6356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:48.299941 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:48.299949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:48.299953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:48.300188 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:48.300755 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:48.300761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:48.300765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:48.301212 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:48.302967 (+ 1755us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:48.303135 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 22:04:48.303422 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":5371,"thread_start_us":5199,"threads_started":1}
I20260927 22:04:48.304869 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.265627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:40523 (local address 127.3.159.62:40619)
0927 22:04:48.269985 (+ 4358us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:48.269996 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:48.270044 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:04:48.272041 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:48.272050 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:48.272199 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:48.272365 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:48.276070 (+ 3705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.277649 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.280708 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.281603 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.281785 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.294517 (+ 12732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:48.294546 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:48.294551 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:48.294643 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:48.299346 (+ 4703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:48.300968 (+ 1622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:48.300974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:48.300977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:48.301198 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:48.302138 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:48.302144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:48.302148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:48.302590 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:48.302740 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:48.304092 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0927 22:04:48.304380 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4129,"thread_start_us":3955,"threads_started":1}
I20260927 22:04:48.309617 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.262836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:40619 (local address 127.3.159.3:40523)
0927 22:04:48.266393 (+ 3557us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:48.266478 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:48.270216 (+ 3738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:48.273519 (+ 3303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:48.273537 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:48.274565 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:48.275629 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.275670 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.277982 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.277989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.280352 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.280381 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.282260 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.282269 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.282637 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.288395 (+ 5758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:48.288450 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:48.293999 (+ 5549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:48.299868 (+ 5869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:48.299884 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:48.299914 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:48.300746 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:48.301369 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:48.301430 (+ 61us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:48.301435 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:48.301974 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:48.302911 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:48.302922 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:48.303541 (+ 619us) client_negotiation.cc:769] Sending connection context
0927 22:04:48.306455 (+ 2914us) client_negotiation.cc:240] Negotiation successful
0927 22:04:48.307428 (+ 973us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3050,"thread_start_us":7603,"threads_started":1}
I20260927 22:04:48.311863 17119 heartbeater.cc:347] Connected to a master server at 127.3.159.62:40619
I20260927 22:04:48.312424 17119 heartbeater.cc:464] Registering TS with master...
I20260927 22:04:48.313798 17119 heartbeater.cc:511] Master 127.3.159.62:40619 requested a full tablet report, sending...
I20260927 22:04:48.316982 16636 ts_manager.cc:194] Registered new tserver with Master: df8da39e95ed4f53aeacc85a5c452fbe (127.3.159.3:34843)
I20260927 22:04:48.318305 16636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:40523
I20260927 22:04:48.321913 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:04:48.349901 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.326052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.3.159.62:40619)
0927 22:04:48.326310 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:48.326317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:48.327127 (+ 810us) server_negotiation.cc:407] Connection header received
0927 22:04:48.327365 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:48.327371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:48.327498 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:48.327633 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:48.330364 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.331662 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.334229 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.334940 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.335095 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.339436 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:48.339445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:48.339451 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:48.339520 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:48.344505 (+ 4985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:48.345879 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:48.345886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:48.345890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:48.346128 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:48.346808 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:48.346816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:48.346820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:48.347311 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:48.347608 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:48.349104 (+ 1496us) server_negotiation.cc:299] Negotiation successful
0927 22:04:48.349464 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 22:04:48.363667 16636 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:04:48.377300 16636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:04:48.427043 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.401961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:34843 (local address 127.0.0.1:40298)
0927 22:04:48.403094 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:48.403149 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:48.403411 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:48.405003 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:48.405013 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:48.405051 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:04:48.405559 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.405575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.411118 (+ 5543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.411128 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.415131 (+ 4003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.415176 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.425716 (+ 10540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.425725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.426055 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.426147 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 22:04:48.426308 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 22:04:48.426584 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":1192,"threads_started":1}
I20260927 22:04:48.429760 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.402871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.3.159.3:34843)
0927 22:04:48.404196 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:48.404207 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:48.404250 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:04:48.404368 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:48.404379 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:48.404694 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:48.405793 (+ 1099us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:04:48.406149 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.408508 (+ 2359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.415572 (+ 7064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.417318 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.428009 (+ 10691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.428502 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:48.428623 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 22:04:48.429032 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":2404,"threads_started":1}
I20260927 22:04:48.435489 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.404297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35163 (local address 127.0.0.1:51266)
0927 22:04:48.410505 (+ 6208us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:48.410540 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:48.410834 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:48.419064 (+ 8230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:48.419075 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:48.419133 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:04:48.419758 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.419778 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.423489 (+ 3711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.423511 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.431772 (+ 8261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.431807 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.434503 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.434511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.434723 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.434763 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:04:48.434871 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 22:04:48.435050 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5979,"thread_start_us":5767,"threads_started":1}
I20260927 22:04:48.436461 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.405043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.3.159.1:35163)
0927 22:04:48.409707 (+ 4664us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:48.409725 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:48.417657 (+ 7932us) server_negotiation.cc:407] Connection header received
0927 22:04:48.417858 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:48.417872 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:48.418318 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:48.418648 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:04:48.421337 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.423231 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.432113 (+ 8882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.433843 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.433998 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.434309 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:48.435873 (+ 1564us) server_negotiation.cc:299] Negotiation successful
0927 22:04:48.436083 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4218,"thread_start_us":3848,"threads_started":1}
I20260927 22:04:48.436010 17047 tablet_service.cc:1467] Processing CreateTablet for tablet df27b9901b6741d5af0210b8b19d8e3f (DEFAULT_TABLE table=test-table [id=f5eb9f15382a4f10a55f25ce0e1ce47d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:04:48.439615 17047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df27b9901b6741d5af0210b8b19d8e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:04:48.442340 16768 tablet_service.cc:1467] Processing CreateTablet for tablet df27b9901b6741d5af0210b8b19d8e3f (DEFAULT_TABLE table=test-table [id=f5eb9f15382a4f10a55f25ce0e1ce47d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:04:48.445906 16768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df27b9901b6741d5af0210b8b19d8e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:04:48.462781 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.406786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.3.159.2:33665)
0927 22:04:48.422447 (+ 15661us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:48.422458 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:48.422495 (+ 37us) server_negotiation.cc:407] Connection header received
0927 22:04:48.422625 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:48.422636 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:48.422984 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:48.423294 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:04:48.433881 (+ 10587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.435559 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.445802 (+ 10243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.447411 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.447580 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.447844 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:48.461792 (+ 13948us) server_negotiation.cc:299] Negotiation successful
0927 22:04:48.462130 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15428,"thread_start_us":16743,"threads_started":1}
I20260927 22:04:48.464195 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.406247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:33665 (local address 127.0.0.1:44552)
0927 22:04:48.420796 (+ 14549us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:48.420842 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:48.421032 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:48.423326 (+ 2294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:48.423333 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:48.423385 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:04:48.424259 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.424279 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.442948 (+ 18669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.442956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.445550 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.445579 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.449618 (+ 4039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.449627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.449848 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.449894 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:04:48.463239 (+ 13345us) client_negotiation.cc:240] Negotiation successful
0927 22:04:48.463590 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14275,"thread_start_us":13982,"threads_started":1}
I20260927 22:04:48.465461 17140 tablet_bootstrap.cc:492] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe: Bootstrap starting.
I20260927 22:04:48.470721 17140 tablet_bootstrap.cc:654] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe: Neither blocks nor log segments found. Creating new log.
I20260927 22:04:48.471168 16913 tablet_service.cc:1467] Processing CreateTablet for tablet df27b9901b6741d5af0210b8b19d8e3f (DEFAULT_TABLE table=test-table [id=f5eb9f15382a4f10a55f25ce0e1ce47d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:04:48.472254 17140 log.cc:824] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe: Log is configured to *not* fsync() on all Append() calls
I20260927 22:04:48.473882 16913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df27b9901b6741d5af0210b8b19d8e3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:04:48.484402 17142 tablet_bootstrap.cc:492] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334: Bootstrap starting.
I20260927 22:04:48.493970 17140 tablet_bootstrap.cc:492] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe: No bootstrap required, opened a new log
I20260927 22:04:48.494359 17140 ts_tablet_manager.cc:1397] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.014s
I20260927 22:04:48.496791 17142 tablet_bootstrap.cc:654] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334: Neither blocks nor log segments found. Creating new log.
I20260927 22:04:48.498755 17142 log.cc:824] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334: Log is configured to *not* fsync() on all Append() calls
I20260927 22:04:48.511444 17140 raft_consensus.cc:348] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34843 } } peers { permanent_uuid: "8e3d2b0688114625bfd477d42fc7f334" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35163 } } peers { permanent_uuid: "2bf311513186424aa56c8a1a812f73d4" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33665 } }
I20260927 22:04:48.512014 17140 raft_consensus.cc:374] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:04:48.512122 17140 raft_consensus.cc:729] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8da39e95ed4f53aeacc85a5c452fbe, State: Initialized, Role: FOLLOWER
I20260927 22:04:48.513006 17142 tablet_bootstrap.cc:492] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334: No bootstrap required, opened a new log
I20260927 22:04:48.514410 17140 consensus_queue.cc:260] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34843 } } peers { permanent_uuid: "8e3d2b0688114625bfd477d42fc7f334" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35163 } } peers { permanent_uuid: "2bf311513186424aa56c8a1a812f73d4" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33665 } }
I20260927 22:04:48.515965 17142 ts_tablet_manager.cc:1397] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334: Time spent bootstrapping tablet: real 0.032s user 0.018s sys 0.000s
I20260927 22:04:48.524753 17144 tablet_bootstrap.cc:492] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4: Bootstrap starting.
I20260927 22:04:48.525271 17140 ts_tablet_manager.cc:1428] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe: Time spent starting tablet: real 0.031s user 0.014s sys 0.006s
I20260927 22:04:48.526070 17119 heartbeater.cc:503] Master 127.3.159.62:40619 was elected leader, sending a full tablet report...
I20260927 22:04:48.532238 17144 tablet_bootstrap.cc:654] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4: Neither blocks nor log segments found. Creating new log.
I20260927 22:04:48.533185 17142 raft_consensus.cc:348] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34843 } } peers { permanent_uuid: "8e3d2b0688114625bfd477d42fc7f334" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35163 } } peers { permanent_uuid: "2bf311513186424aa56c8a1a812f73d4" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33665 } }
I20260927 22:04:48.534052 17142 raft_consensus.cc:374] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:04:48.534206 17142 raft_consensus.cc:729] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3d2b0688114625bfd477d42fc7f334, State: Initialized, Role: FOLLOWER
I20260927 22:04:48.534421 17144 log.cc:824] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:04:48.535224 17142 consensus_queue.cc:260] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34843 } } peers { permanent_uuid: "8e3d2b0688114625bfd477d42fc7f334" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35163 } } peers { permanent_uuid: "2bf311513186424aa56c8a1a812f73d4" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33665 } }
I20260927 22:04:48.545037 17142 ts_tablet_manager.cc:1428] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334: Time spent starting tablet: real 0.029s user 0.014s sys 0.008s
I20260927 22:04:48.548789 17144 tablet_bootstrap.cc:492] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4: No bootstrap required, opened a new log
I20260927 22:04:48.549342 17144 ts_tablet_manager.cc:1397] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.012s
I20260927 22:04:48.560573 17144 raft_consensus.cc:348] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34843 } } peers { permanent_uuid: "8e3d2b0688114625bfd477d42fc7f334" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35163 } } peers { permanent_uuid: "2bf311513186424aa56c8a1a812f73d4" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33665 } }
I20260927 22:04:48.561146 17144 raft_consensus.cc:374] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:04:48.561260 17144 raft_consensus.cc:729] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf311513186424aa56c8a1a812f73d4, State: Initialized, Role: FOLLOWER
I20260927 22:04:48.562022 17144 consensus_queue.cc:260] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34843 } } peers { permanent_uuid: "8e3d2b0688114625bfd477d42fc7f334" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35163 } } peers { permanent_uuid: "2bf311513186424aa56c8a1a812f73d4" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33665 } }
I20260927 22:04:48.564155 17144 ts_tablet_manager.cc:1428] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260927 22:04:48.572300 17148 raft_consensus.cc:482] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:04:48.572715 17148 raft_consensus.cc:504] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34843 } } peers { permanent_uuid: "8e3d2b0688114625bfd477d42fc7f334" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35163 } } peers { permanent_uuid: "2bf311513186424aa56c8a1a812f73d4" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33665 } }
I20260927 22:04:48.581100 17148 leader_election.cc:288] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df8da39e95ed4f53aeacc85a5c452fbe (127.3.159.3:34843), 8e3d2b0688114625bfd477d42fc7f334 (127.3.159.1:35163)
I20260927 22:04:48.585160 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.576244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:50757 (local address 127.3.159.3:34843)
0927 22:04:48.576503 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:48.576510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:48.577329 (+ 819us) server_negotiation.cc:407] Connection header received
0927 22:04:48.577692 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:48.577698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:48.577776 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:48.577840 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:04:48.578687 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.579757 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.582635 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.583826 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.583951 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.584076 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:48.584751 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 22:04:48.584901 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:04:48.586344 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.575590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:34843 (local address 127.3.159.2:50757)
0927 22:04:48.577212 (+ 1622us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:48.577236 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:48.577498 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:48.578049 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:48.578054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:48.578079 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:04:48.578501 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.578515 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.579956 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.579962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.582385 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.582408 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.584319 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.584326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.584530 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.584576 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:04:48.585745 (+ 1169us) client_negotiation.cc:240] Negotiation successful
0927 22:04:48.585913 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1468,"thread_start_us":1255,"threads_started":1}
I20260927 22:04:48.588274 17074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27b9901b6741d5af0210b8b19d8e3f" candidate_uuid: "2bf311513186424aa56c8a1a812f73d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" is_pre_election: true
I20260927 22:04:48.589051 17074 raft_consensus.cc:2454] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf311513186424aa56c8a1a812f73d4 in term 0.
I20260927 22:04:48.590088 16869 leader_election.cc:302] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bf311513186424aa56c8a1a812f73d4, df8da39e95ed4f53aeacc85a5c452fbe; no voters:
I20260927 22:04:48.590662 17148 raft_consensus.cc:2781] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:04:48.590816 17148 raft_consensus.cc:482] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:04:48.590988 17148 raft_consensus.cc:3037] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:04:48.594823 17148 raft_consensus.cc:504] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34843 } } peers { permanent_uuid: "8e3d2b0688114625bfd477d42fc7f334" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35163 } } peers { permanent_uuid: "2bf311513186424aa56c8a1a812f73d4" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33665 } }
I20260927 22:04:48.595741 17148 leader_election.cc:288] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [CANDIDATE]: Term 1 election: Requested vote from peers df8da39e95ed4f53aeacc85a5c452fbe (127.3.159.3:34843), 8e3d2b0688114625bfd477d42fc7f334 (127.3.159.1:35163)
I20260927 22:04:48.596752 17074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27b9901b6741d5af0210b8b19d8e3f" candidate_uuid: "2bf311513186424aa56c8a1a812f73d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8da39e95ed4f53aeacc85a5c452fbe"
I20260927 22:04:48.596863 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.578140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:37409 (local address 127.3.159.1:35163)
0927 22:04:48.578415 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:48.578425 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:48.586991 (+ 8566us) server_negotiation.cc:407] Connection header received
0927 22:04:48.587356 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:48.587363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:48.587456 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:48.587529 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:04:48.588500 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.589618 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.593441 (+ 3823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.595042 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.595232 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.595359 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:48.596395 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0927 22:04:48.596567 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 22:04:48.597018 17074 raft_consensus.cc:3037] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:04:48.597959 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.577651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35163 (local address 127.3.159.2:37409)
0927 22:04:48.586710 (+ 9059us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:48.586742 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:48.587161 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:48.587794 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:48.587800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:48.587836 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:04:48.588210 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.588227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.590071 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.590079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.593075 (+ 2996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:48.593108 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.595927 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.595937 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:48.596158 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.596194 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:04:48.597432 (+ 1238us) client_negotiation.cc:240] Negotiation successful
0927 22:04:48.597604 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8866,"thread_start_us":9121,"threads_started":1}
I20260927 22:04:48.599658 16787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27b9901b6741d5af0210b8b19d8e3f" candidate_uuid: "2bf311513186424aa56c8a1a812f73d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e3d2b0688114625bfd477d42fc7f334"
I20260927 22:04:48.599637 16788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27b9901b6741d5af0210b8b19d8e3f" candidate_uuid: "2bf311513186424aa56c8a1a812f73d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e3d2b0688114625bfd477d42fc7f334" is_pre_election: true
I20260927 22:04:48.600332 16788 raft_consensus.cc:2454] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bf311513186424aa56c8a1a812f73d4 in term 0.
I20260927 22:04:48.600835 17074 raft_consensus.cc:2454] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bf311513186424aa56c8a1a812f73d4 in term 1.
I20260927 22:04:48.601893 16869 leader_election.cc:302] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2bf311513186424aa56c8a1a812f73d4, df8da39e95ed4f53aeacc85a5c452fbe; no voters: 8e3d2b0688114625bfd477d42fc7f334
I20260927 22:04:48.602341 17148 raft_consensus.cc:2781] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:04:48.603027 17148 raft_consensus.cc:686] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf311513186424aa56c8a1a812f73d4, State: Running, Role: LEADER
I20260927 22:04:48.603727 17148 consensus_queue.cc:237] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34843 } } peers { permanent_uuid: "8e3d2b0688114625bfd477d42fc7f334" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35163 } } peers { permanent_uuid: "2bf311513186424aa56c8a1a812f73d4" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33665 } }
I20260927 22:04:48.614221 16635 catalog_manager.cc:5495] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bf311513186424aa56c8a1a812f73d4 (127.3.159.2). New cstate: current_term: 1 leader_uuid: "2bf311513186424aa56c8a1a812f73d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e3d2b0688114625bfd477d42fc7f334" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bf311513186424aa56c8a1a812f73d4" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 33665 } health_report { overall_health: HEALTHY } } }
W20260927 22:04:48.628706 16834 tablet.cc:2369] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:04:48.665143 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.656134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.3.159.2:33665)
0927 22:04:48.656438 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:48.656446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:48.656596 (+ 150us) server_negotiation.cc:407] Connection header received
0927 22:04:48.656946 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:48.656953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:48.657107 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:48.657320 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:04:48.657872 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.659084 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.661293 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.662226 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.662418 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.663165 (+ 747us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:04:48.664031 (+ 866us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:04:48.664222 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:48.664400 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 22:04:48.664543 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 22:04:48.677843 16788 raft_consensus.cc:3037] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:04:48.678213 17074 raft_consensus.cc:1261] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe [term 1 FOLLOWER]: Refusing update from remote peer 2bf311513186424aa56c8a1a812f73d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:04:48.679601 17148 consensus_queue.cc:1035] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8da39e95ed4f53aeacc85a5c452fbe" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:04:48.682111 16788 raft_consensus.cc:1261] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334 [term 1 FOLLOWER]: Refusing update from remote peer 2bf311513186424aa56c8a1a812f73d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:04:48.684335 17148 consensus_queue.cc:1035] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e3d2b0688114625bfd477d42fc7f334" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:04:48.716360 17154 mvcc.cc:205] Tried to move back new op lower bound from 7334079236810747904 to 7334079236526780416. Current Snapshot: MvccSnapshot[applied={T|T < 7334079236810747904}]
I20260927 22:04:48.719365 17156 mvcc.cc:205] Tried to move back new op lower bound from 7334079236810747904 to 7334079236526780416. Current Snapshot: MvccSnapshot[applied={T|T < 7334079236810747904}]
I20260927 22:04:48.729784 17155 mvcc.cc:205] Tried to move back new op lower bound from 7334079236810747904 to 7334079236526780416. Current Snapshot: MvccSnapshot[applied={T|T < 7334079236810747904}]
I20260927 22:04:48.752472 16635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:04:48.752841 16635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:04:48.757534 16635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98a53bfca55c4f78bd8d2f43fe91e95f: Sending DeleteTablet for 3 replicas of tablet df27b9901b6741d5af0210b8b19d8e3f
I20260927 22:04:48.759291 16913 tablet_service.cc:1514] Processing DeleteTablet for tablet df27b9901b6741d5af0210b8b19d8e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:04:48 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:44552
I20260927 22:04:48.759450 17047 tablet_service.cc:1514] Processing DeleteTablet for tablet df27b9901b6741d5af0210b8b19d8e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:04:48 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:40298
I20260927 22:04:48.759815 16768 tablet_service.cc:1514] Processing DeleteTablet for tablet df27b9901b6741d5af0210b8b19d8e3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:04:48 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:51266
I20260927 22:04:48.769155 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 16673
I20260927 22:04:48.772399 17163 tablet_replica.cc:330] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4: stopping tablet replica
I20260927 22:04:48.772902 17163 raft_consensus.cc:2229] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:04:48.773686 17163 raft_consensus.cc:2258] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:04:48.778395 17163 ts_tablet_manager.cc:1905] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:04:48.790261 17165 tablet_replica.cc:330] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334: stopping tablet replica
I20260927 22:04:48.791416 17164 tablet_replica.cc:330] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe: stopping tablet replica
I20260927 22:04:48.792099 17165 raft_consensus.cc:2229] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:04:48.792600 17164 raft_consensus.cc:2229] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:04:48.793196 17164 raft_consensus.cc:2258] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:04:48.793763 17165 raft_consensus.cc:2258] T df27b9901b6741d5af0210b8b19d8e3f P 8e3d2b0688114625bfd477d42fc7f334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:04:48.796434 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:48.771445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51270 (local address 127.3.159.1:35163)
0927 22:04:48.771672 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:48.771681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:48.772007 (+ 326us) server_negotiation.cc:407] Connection header received
0927 22:04:48.772285 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:48.772293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:48.772434 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:48.772584 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:48.776504 (+ 3920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.777780 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.781222 (+ 3442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:48.781994 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:48.782203 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:48.786751 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:48.786779 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:48.786806 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:48.786884 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:48.791638 (+ 4754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:48.792848 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:48.792860 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:48.792864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:48.793097 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:48.793627 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:48.793632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:48.793636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:48.794668 (+ 1032us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:48.795012 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:48.795773 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 22:04:48.796037 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 22:04:48.797202 16828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:48.798794 17164 ts_tablet_manager.cc:1905] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:04:48.804338 17163 ts_tablet_manager.cc:1918] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:04:48.809471 17163 log.cc:1196] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/df27b9901b6741d5af0210b8b19d8e3f
I20260927 22:04:48.810212 17163 ts_tablet_manager.cc:1939] T df27b9901b6741d5af0210b8b19d8e3f P 2bf311513186424aa56c8a1a812f73d4: Deleting consensus metadata
I20260927 22:04:48.815451 17164 ts_tablet_manager.cc:1918] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:04:48.815682 16622 catalog_manager.cc:4841] TS 2bf311513186424aa56c8a1a812f73d4 (127.3.159.2:33665): tablet df27b9901b6741d5af0210b8b19d8e3f (table test-table [id=f5eb9f15382a4f10a55f25ce0e1ce47d]) successfully deleted
I20260927 22:04:48.815884 17164 log.cc:1196] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/df27b9901b6741d5af0210b8b19d8e3f
I20260927 22:04:48.816524 17164 ts_tablet_manager.cc:1939] T df27b9901b6741d5af0210b8b19d8e3f P df8da39e95ed4f53aeacc85a5c452fbe: Deleting consensus metadata
I20260927 22:04:48.829187 16623 catalog_manager.cc:4841] TS df8da39e95ed4f53aeacc85a5c452fbe (127.3.159.3:34843): tablet df27b9901b6741d5af0210b8b19d8e3f (table test-table [id=f5eb9f15382a4f10a55f25ce0e1ce47d]) successfully deleted
I20260927 22:04:48.976204 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 16673
W20260927 22:04:49.003216 16623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.159.1:35163 (error 108)
I20260927 22:04:49.009693 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 16837
I20260927 22:04:49.009793 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:49.008513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35163 (local address 127.0.0.1:51278)
0927 22:04:49.009104 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:49.009273 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":271,"threads_started":1}
W20260927 22:04:49.010443 16623 catalog_manager.cc:4584] TS 8e3d2b0688114625bfd477d42fc7f334 (127.3.159.1:35163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet df27b9901b6741d5af0210b8b19d8e3f: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35163: connect: Connection refused (error 111)
I20260927 22:04:49.036055 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:49.013124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.3.159.2:33665)
0927 22:04:49.013456 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:49.013463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:49.013497 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:04:49.013599 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:49.013606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:49.013712 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:49.013880 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:49.016359 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:49.017732 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:49.020109 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:49.020891 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:49.021104 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:49.025316 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:49.025339 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:49.025364 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:49.025463 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:49.030813 (+ 5350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:49.032681 (+ 1868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:49.032695 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:49.032700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:49.032971 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:49.033634 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:49.033641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:49.033645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:49.034229 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:49.034600 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:49.035020 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 22:04:49.035355 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 22:04:49.036875 16973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:49.065023 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:49.064005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35163 (local address 127.0.0.1:51290)
0927 22:04:49.064617 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:49.064811 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260927 22:04:49.066938 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:49.066182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35163 (local address 127.0.0.1:51306)
0927 22:04:49.066432 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:49.066562 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:04:49.128552 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:49.127836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35163 (local address 127.0.0.1:51308)
0927 22:04:49.128178 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:49.128329 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260927 22:04:49.130434 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:49.129802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35163 (local address 127.0.0.1:51324)
0927 22:04:49.130052 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:49.130180 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 22:04:49.172595 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 16837
I20260927 22:04:49.193619 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 16982
I20260927 22:04:49.211803 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:49.211148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35163 (local address 127.0.0.1:51334)
0927 22:04:49.211435 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:49.211539 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 22:04:49.213109 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:49.194719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.3.159.3:34843)
0927 22:04:49.195261 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:49.195266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:49.195289 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:04:49.195350 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:49.195353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:49.195420 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:49.195529 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:49.197270 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:49.198120 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:49.199857 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:49.200351 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:49.200508 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:49.204400 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:49.204424 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:49.204447 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:49.204510 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:49.208927 (+ 4417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:49.210108 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:49.210119 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:49.210121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:49.210339 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:49.210873 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:49.210879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:49.210882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:49.211506 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:49.211785 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:49.212374 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 22:04:49.212660 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":299,"threads_started":1}
I20260927 22:04:49.213130 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:49.212594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35163 (local address 127.0.0.1:51342)
0927 22:04:49.212863 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:49.212945 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260927 22:04:49.213974 17114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:49.320770 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 16982
I20260927 22:04:49.340230 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 16603
I20260927 22:04:49.341732 16666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:49.478714 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:49.477988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35163 (local address 127.0.0.1:51346)
0927 22:04:49.478334 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:49.478457 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260927 22:04:49.479066 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 16603
I20260927 22:04:49.505980 3708 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 22:04:49.853792 3708 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 22:04:49.854010 3708 mini_ranger.cc:66] Stopping Ranger...
I20260927 22:04:50.202777 3708 mini_ranger.cc:68] Stopped Ranger
2026-09-27 22:04:50.203 UTC [15655] LOG: received smart shutdown request
2026-09-27 22:04:50.208 UTC [15655] LOG: background worker "logical replication launcher" (PID 15664) exited with exit code 1
2026-09-27 22:04:50.209 UTC [15659] LOG: shutting down
2026-09-27 22:04:50.517 UTC [15655] LOG: database system is shut down
2026-09-27T22:04:50Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (70076 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:04:50 dist-test-slave-sg4j krb5kdc[17181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:04:50 dist-test-slave-sg4j krb5kdc[17181](info): set up 2 sockets
Sep 27 22:04:50 dist-test-slave-sg4j krb5kdc[17181](info): commencing operation
krb5kdc: starting...
W20260927 22:04:51.825981 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:04:52 dist-test-slave-sg4j krb5kdc[17181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:04:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:04:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:04:52.119367 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:43719
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:43719,127.3.159.61:37583,127.3.159.60:38999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:04:52.544942 17194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:52.545455 17194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:52.545626 17194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:52.554584 17194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:04:52.554754 17194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:52.554807 17194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:52.555033 17194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:04:52.555143 17194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:04:52.568814 17194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:43719,127.3.159.61:37583,127.3.159.60:38999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:43719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:52.570997 17194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:52.572947 17194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:52.590086 17199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:52.590032 17200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:52.589987 17202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:52.591316 17194 server_base.cc:1034] running on GCE node
I20260927 22:04:52.592525 17194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:52.594816 17194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:52.596161 17194 hybrid_clock.cc:648] HybridClock initialized: now 1790546692596117 us; error 125 us; skew 500 ppm
I20260927 22:04:52.596726 17194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:52 dist-test-slave-sg4j krb5kdc[17181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:52.601576 17194 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:04:52.604079 17194 webserver.cc:460] Webserver started at http://127.3.159.62:41087/ using document root <none> and password file <none>
I20260927 22:04:52.604967 17194 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:52.605104 17194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:04:52.605526 17194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:04:52.609268 17194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "fc9eda4de8694173b290ad70307810b8"
format_stamp: "Formatted at 2026-09-27 22:04:52 on dist-test-slave-sg4j"
server_key: "33e3fed3f2e96cad0ecd11f526809cc4"
server_key_iv: "9f329fd9289d34ac225abbfbd14c16f1"
server_key_version: "encryptionkey@0"
I20260927 22:04:52.610773 17194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "fc9eda4de8694173b290ad70307810b8"
format_stamp: "Formatted at 2026-09-27 22:04:52 on dist-test-slave-sg4j"
server_key: "33e3fed3f2e96cad0ecd11f526809cc4"
server_key_iv: "9f329fd9289d34ac225abbfbd14c16f1"
server_key_version: "encryptionkey@0"
I20260927 22:04:52.620558 17194 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260927 22:04:52.625562 17209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:52.627616 17194 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:04:52.627883 17194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "fc9eda4de8694173b290ad70307810b8"
format_stamp: "Formatted at 2026-09-27 22:04:52 on dist-test-slave-sg4j"
server_key: "33e3fed3f2e96cad0ecd11f526809cc4"
server_key_iv: "9f329fd9289d34ac225abbfbd14c16f1"
server_key_version: "encryptionkey@0"
I20260927 22:04:52.628132 17194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:04:52.652196 17194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:52.660295 17194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:52.660737 17194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:52.692028 17260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:43719 every 8 connection(s)
I20260927 22:04:52.692028 17194 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:43719
I20260927 22:04:52.694414 17194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:04:52.700202 17261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:04:52.703713 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 17194
I20260927 22:04:52.704027 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:04:52.704964 3708 external_mini_cluster.cc:1428] Setting key 19c9d4f9d8c3468724e73bdf0caab6ee
I20260927 22:04:52.712981 17261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.159.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.61@KRBTEST.COM" created.
Sep 27 22:04:52 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:04:52.756075 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:52.719442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:43719 (local address 127.0.0.1:60756)
0927 22:04:52.721660 (+ 2218us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:52.721734 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:52.725089 (+ 3355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:52.726824 (+ 1735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:52.726837 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:52.727881 (+ 1044us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:52.728550 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:52.728586 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:52.730875 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:52.730886 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:52.734031 (+ 3145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:52.734067 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:52.735808 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:52.735823 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:52.736121 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:52.737519 (+ 1398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:52.737606 (+ 87us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:52.745020 (+ 7414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:52.749532 (+ 4512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:52.749560 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:52.749582 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:52.750501 (+ 919us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:52.751541 (+ 1040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:52.751550 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:52.751557 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:52.751895 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:52.753233 (+ 1338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:52.753241 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:52.753592 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 22:04:52.753938 (+ 346us) client_negotiation.cc:240] Negotiation successful
0927 22:04:52.754571 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1686,"thread_start_us":1355,"threads_started":1}
I20260927 22:04:52.756146 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:52.719517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.3.159.62:43719)
0927 22:04:52.721121 (+ 1604us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:52.721150 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:52.721922 (+ 772us) server_negotiation.cc:407] Connection header received
0927 22:04:52.725876 (+ 3954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:52.725885 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:52.726368 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:52.726638 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:52.728912 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:52.730565 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:52.734418 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:52.735339 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:52.735571 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:52.745605 (+ 10034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:52.745618 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:52.745637 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:52.745728 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:52.749206 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:52.751047 (+ 1841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:52.751057 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:52.751060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:52.751360 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:52.752095 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:52.752103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:52.752106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:52.753020 (+ 914us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:52.753299 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:52.754190 (+ 891us) server_negotiation.cc:299] Negotiation successful
0927 22:04:52.754830 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":202,"server-negotiator.queue_time_us":1148,"thread_start_us":796,"threads_started":1}
I20260927 22:04:52.760535 17261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:04:52.762462 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:52.761523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57784)
0927 22:04:52.762031 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:52.762245 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260927 22:04:52.762943 17213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
I20260927 22:04:52.765542 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:52.764790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57796)
0927 22:04:52.765078 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:52.765224 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 22:04:52.765908 17261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:37583: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab.
Entry for principal kudu/127.3.159.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab.
I20260927 22:04:52.783536 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.61:37583
--webserver_interface=127.3.159.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab
--principal=kudu/127.3.159.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:43719,127.3.159.61:37583,127.3.159.60:38999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:04:52.815368 17261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } attempt: 1
I20260927 22:04:52.816936 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:52.816304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57804)
0927 22:04:52.816597 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:52.816751 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 22:04:52.818274 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:52.817791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57810)
0927 22:04:52.817967 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:52.818067 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 22:04:52.818565 17261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:37583: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
I20260927 22:04:52.886969 17261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } attempt: 2
I20260927 22:04:52.888584 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:52.887911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57814)
0927 22:04:52.888225 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:52.888375 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 22:04:52.890017 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:52.889528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57818)
0927 22:04:52.889695 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:52.889827 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:04:52.890287 17261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:37583: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
I20260927 22:04:52.981690 17261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } attempt: 3
I20260927 22:04:52.983266 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:52.982643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57830)
0927 22:04:52.982981 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:52.983081 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260927 22:04:52.984833 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:52.984202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57840)
0927 22:04:52.984429 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:52.984580 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 22:04:52.985073 17261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:37583: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
I20260927 22:04:53.128389 17261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } attempt: 4
I20260927 22:04:53.130234 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.129479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57856)
0927 22:04:53.129891 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.130039 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260927 22:04:53.131881 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.131307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57864)
0927 22:04:53.131509 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.131644 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 22:04:53.132193 17261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:37583: Network error: Client connection negotiation failed: client connection to 127.3.159.61:37583: connect: Connection refused (error 111)
W20260927 22:04:53.189661 17268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:53.190017 17268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:53.190173 17268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:53.200531 17268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:04:53.200693 17268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:53.200744 17268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:53.200857 17268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:04:53.201285 17268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:04:53.215684 17268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:43719,127.3.159.61:37583,127.3.159.60:38999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab
--principal=kudu/127.3.159.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.61:37583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.159.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:53.217294 17268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:53.218803 17268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:53.234119 17276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:53.234202 17274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:53.234613 17273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:53.236533 17268 server_base.cc:1034] running on GCE node
I20260927 22:04:53.237643 17268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:53.239486 17268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:53.241110 17268 hybrid_clock.cc:648] HybridClock initialized: now 1790546693240979 us; error 107 us; skew 500 ppm
I20260927 22:04:53.241698 17268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:53 dist-test-slave-sg4j krb5kdc[17181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:53.245920 17268 init.cc:377] Logged in from keytab as kudu/127.3.159.61@KRBTEST.COM (short username kudu)
I20260927 22:04:53.247972 17268 webserver.cc:460] Webserver started at http://127.3.159.61:40815/ using document root <none> and password file <none>
I20260927 22:04:53.248785 17268 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:53.248950 17268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:04:53.249317 17268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:04:53.252977 17268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/data/instance:
uuid: "4fc119e7a1c948799b2516438e8c20fb"
format_stamp: "Formatted at 2026-09-27 22:04:53 on dist-test-slave-sg4j"
server_key: "8d8d93b04ede55421db6b9a6ff963258"
server_key_iv: "b43115b10e5f27f2b6fa02e7946aff23"
server_key_version: "encryptionkey@0"
I20260927 22:04:53.254106 17268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/wal/instance:
uuid: "4fc119e7a1c948799b2516438e8c20fb"
format_stamp: "Formatted at 2026-09-27 22:04:53 on dist-test-slave-sg4j"
server_key: "8d8d93b04ede55421db6b9a6ff963258"
server_key_iv: "b43115b10e5f27f2b6fa02e7946aff23"
server_key_version: "encryptionkey@0"
I20260927 22:04:53.262576 17268 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260927 22:04:53.269038 17283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:53.271711 17268 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:04:53.272262 17268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/wal
uuid: "4fc119e7a1c948799b2516438e8c20fb"
format_stamp: "Formatted at 2026-09-27 22:04:53 on dist-test-slave-sg4j"
server_key: "8d8d93b04ede55421db6b9a6ff963258"
server_key_iv: "b43115b10e5f27f2b6fa02e7946aff23"
server_key_version: "encryptionkey@0"
I20260927 22:04:53.272729 17268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:04:53.301601 17268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:53.306699 17268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:53.307029 17268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:53.330427 17268 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.61:37583
I20260927 22:04:53.330540 17334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.61:37583 every 8 connection(s)
I20260927 22:04:53.332481 17268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
I20260927 22:04:53.336365 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 17268
I20260927 22:04:53.336622 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-1/wal/instance
I20260927 22:04:53.337288 3708 external_mini_cluster.cc:1428] Setting key a7a7b99a64f47f68379c938cd5bc1872
I20260927 22:04:53.339879 17335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:04:53.351747 17335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.159.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.60@KRBTEST.COM" created.
Sep 27 22:04:53 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:04:53.386003 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.355710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.3.159.62:43719)
0927 22:04:53.356422 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:53.356427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:53.358058 (+ 1631us) server_negotiation.cc:407] Connection header received
0927 22:04:53.360826 (+ 2768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:53.360837 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:53.360995 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:53.361168 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:53.363720 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.365230 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.368034 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.368987 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.369215 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:53.376050 (+ 6835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:53.376060 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:53.376068 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:53.376137 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:53.380719 (+ 4582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:53.382167 (+ 1448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:53.382180 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:53.382187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:53.382470 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:53.383137 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:53.383146 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:53.383153 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:53.383694 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:53.383900 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:53.384949 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0927 22:04:53.385329 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":398,"threads_started":1}
I20260927 22:04:53.386826 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.356401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:43719 (local address 127.0.0.1:60762)
0927 22:04:53.357751 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.357879 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:53.360262 (+ 2383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:53.361657 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:53.361671 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:53.362509 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:53.363387 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:53.363417 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.365629 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.365635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:53.367660 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:53.367689 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.369460 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.369503 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:53.370013 (+ 510us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:53.371335 (+ 1322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:53.371384 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:53.375647 (+ 4263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:53.381005 (+ 5358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:53.381015 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:53.381031 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:53.381906 (+ 875us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:53.382680 (+ 774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:53.382685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:53.382688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:53.382958 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:53.383857 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:53.383921 (+ 64us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:53.384473 (+ 552us) client_negotiation.cc:769] Sending connection context
0927 22:04:53.384747 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 22:04:53.385272 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":581,"threads_started":1}
I20260927 22:04:53.389480 17335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:04:53 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab.
Entry for principal kudu/127.3.159.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab.
I20260927 22:04:53.409451 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.60:38999
--webserver_interface=127.3.159.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab
--principal=kudu/127.3.159.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:43719,127.3.159.61:37583,127.3.159.60:38999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:04:53.411077 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.390516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57880)
0927 22:04:53.390723 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.390748 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:53.390908 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:53.392494 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:53.392504 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:53.393231 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:53.393590 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:53.393612 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.395329 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.395340 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:53.397184 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:53.397214 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.398435 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.398444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:53.398718 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:53.399575 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:53.399597 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:53.402834 (+ 3237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:53.407383 (+ 4549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:53.407389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:53.407392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:53.408068 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:53.408675 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:53.408681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:53.408684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:53.408872 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:53.409944 (+ 1072us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:53.409948 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:53.410117 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 22:04:53.410266 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:04:53.410547 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:04:53.411579 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.391098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.3.159.61:37583)
0927 22:04:53.391923 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:53.391931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:53.391960 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:04:53.392019 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:53.392025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:53.392229 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:53.392369 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:53.393953 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.395104 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.397528 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.398187 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.398329 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:53.403277 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:53.403297 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:53.403317 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:53.403369 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:53.407064 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:53.408284 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:53.408296 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:53.408301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:53.408511 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:53.409023 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:53.409031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:53.409037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:53.409674 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:53.409994 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:53.410521 (+ 527us) server_negotiation.cc:299] Negotiation successful
0927 22:04:53.410897 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":486,"threads_started":1}
I20260927 22:04:53.414354 17335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:04:53.415971 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.415229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51066)
0927 22:04:53.415536 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.415745 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 22:04:53.416332 17285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
I20260927 22:04:53.418531 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.418042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51068)
0927 22:04:53.418291 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.418379 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260927 22:04:53.419030 17335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38999: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
I20260927 22:04:53.431537 17261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } attempt: 5
Sep 27 22:04:53 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
I20260927 22:04:53.449154 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.432548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57886)
0927 22:04:53.432862 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.432890 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:53.433090 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:53.433650 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:53.433658 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:53.434361 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:53.434694 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:53.434712 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.435936 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.435945 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:53.437797 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:53.437819 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.438866 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.438873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:53.439109 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:53.440147 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:53.440178 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:53.443158 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:53.446522 (+ 3364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:53.446529 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:53.446533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:53.446987 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:53.447392 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:53.447398 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:53.447401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:53.447543 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:53.448088 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:53.448093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:53.448270 (+ 177us) client_negotiation.cc:769] Sending connection context
0927 22:04:53.448450 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 22:04:53.448786 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 22:04:53.449154 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.432845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.3.159.61:37583)
0927 22:04:53.433039 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:53.433044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:53.433069 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:04:53.433302 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:53.433307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:53.433450 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:53.433558 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:53.434977 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.435743 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.438093 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:53.438663 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:53.438799 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:53.443427 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:53.443433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:53.443436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:53.443477 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:53.446319 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:53.447130 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:53.447136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:53.447140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:53.447277 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:53.447655 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:53.447661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:53.447664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:53.447963 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:53.448123 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:53.448550 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 22:04:53.448781 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 22:04:53.450716 17261 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:04:53.452227 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.451559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51082)
0927 22:04:53.451825 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.451981 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:04:53.453558 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.453110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51084)
0927 22:04:53.453274 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.453369 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 22:04:53.454013 17261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38999: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
I20260927 22:04:53.468485 17335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } attempt: 1
I20260927 22:04:53.470088 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.469592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51086)
0927 22:04:53.469801 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.469917 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 22:04:53.471474 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.471052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51098)
0927 22:04:53.471230 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.471341 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 22:04:53.471694 17335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38999: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
I20260927 22:04:53.505345 17261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } attempt: 1
I20260927 22:04:53.507009 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.506119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51100)
0927 22:04:53.506500 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.506690 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260927 22:04:53.508612 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.508041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51110)
0927 22:04:53.508280 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.508416 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 22:04:53.508982 17261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38999: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
I20260927 22:04:53.539997 17335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } attempt: 2
I20260927 22:04:53.541755 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.540906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51126)
0927 22:04:53.541259 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.541486 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260927 22:04:53.543345 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.542849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51132)
0927 22:04:53.543040 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.543154 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 22:04:53.543637 17335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38999: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
I20260927 22:04:53.577332 17261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } attempt: 2
I20260927 22:04:53.579030 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.578184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51142)
0927 22:04:53.578579 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.578751 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 22:04:53.580688 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.580077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51148)
0927 22:04:53.580323 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.580479 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260927 22:04:53.581101 17261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38999: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
I20260927 22:04:53.635058 17335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } attempt: 3
I20260927 22:04:53.636801 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.635848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51154)
0927 22:04:53.636243 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.636472 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 22:04:53.638753 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.637968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51166)
0927 22:04:53.638363 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.638530 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
W20260927 22:04:53.639211 17335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38999: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
I20260927 22:04:53.687470 17261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } attempt: 3
I20260927 22:04:53.689489 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.688537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51178)
0927 22:04:53.688908 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.689113 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 22:04:53.691242 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.690539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51182)
0927 22:04:53.690867 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.691016 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260927 22:04:53.691687 17261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38999: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
I20260927 22:04:53.782532 17335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } attempt: 4
I20260927 22:04:53.784205 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.783543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51192)
0927 22:04:53.783864 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.784001 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260927 22:04:53.785895 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.785261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51200)
0927 22:04:53.785525 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.785662 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260927 22:04:53.786224 17335 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38999: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
W20260927 22:04:53.853183 17343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:53.853755 17343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:53.853951 17343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:53.865167 17343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:04:53.865360 17343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:53.865518 17343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:53.865608 17343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:04:53.865662 17343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:04:53.868999 17261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } attempt: 4
I20260927 22:04:53.870682 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.869950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51206)
0927 22:04:53.870261 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.870406 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260927 22:04:53.872254 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:53.871750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51218)
0927 22:04:53.871951 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:53.872066 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 22:04:53.872619 17261 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38999: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38999: connect: Connection refused (error 111)
I20260927 22:04:53.879370 17343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:43719,127.3.159.61:37583,127.3.159.60:38999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab
--principal=kudu/127.3.159.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.60:38999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.159.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:53.881652 17343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:53.883564 17343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:53.899917 17348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:53.901086 17351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:53.901008 17349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:53.902132 17343 server_base.cc:1034] running on GCE node
I20260927 22:04:53.903535 17343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:53.905733 17343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:53.907037 17343 hybrid_clock.cc:648] HybridClock initialized: now 1790546693906967 us; error 78 us; skew 500 ppm
I20260927 22:04:53.907532 17343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:53 dist-test-slave-sg4j krb5kdc[17181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:53.911909 17343 init.cc:377] Logged in from keytab as kudu/127.3.159.60@KRBTEST.COM (short username kudu)
I20260927 22:04:53.914425 17343 webserver.cc:460] Webserver started at http://127.3.159.60:45965/ using document root <none> and password file <none>
I20260927 22:04:53.915519 17343 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:53.915722 17343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:04:53.916158 17343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:04:53.920679 17343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/data/instance:
uuid: "03f358d51483482cb9a2f78e1e21ca03"
format_stamp: "Formatted at 2026-09-27 22:04:53 on dist-test-slave-sg4j"
server_key: "2f602773c21847ba9c1728a6ec619a7b"
server_key_iv: "dea048a637d039519e2e9ecd399c7a5a"
server_key_version: "encryptionkey@0"
I20260927 22:04:53.921926 17343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/wal/instance:
uuid: "03f358d51483482cb9a2f78e1e21ca03"
format_stamp: "Formatted at 2026-09-27 22:04:53 on dist-test-slave-sg4j"
server_key: "2f602773c21847ba9c1728a6ec619a7b"
server_key_iv: "dea048a637d039519e2e9ecd399c7a5a"
server_key_version: "encryptionkey@0"
I20260927 22:04:53.929939 17343 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260927 22:04:53.936044 17358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:53.938506 17343 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 22:04:53.938810 17343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/wal
uuid: "03f358d51483482cb9a2f78e1e21ca03"
format_stamp: "Formatted at 2026-09-27 22:04:53 on dist-test-slave-sg4j"
server_key: "2f602773c21847ba9c1728a6ec619a7b"
server_key_iv: "dea048a637d039519e2e9ecd399c7a5a"
server_key_version: "encryptionkey@0"
I20260927 22:04:53.939190 17343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:04:53.977522 17343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:53.984494 17343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:53.984995 17343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:54.012614 17343 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.60:38999
I20260927 22:04:54.012629 17409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.60:38999 every 8 connection(s)
I20260927 22:04:54.014735 17343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
I20260927 22:04:54.020812 17410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:04:54.024746 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 17343
I20260927 22:04:54.024978 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790546447244210-3708-0/minicluster-data/master-2/wal/instance
I20260927 22:04:54.025650 3708 external_mini_cluster.cc:1428] Setting key 054a0d59e8326d90b63d028cc64bb051
I20260927 22:04:54.032469 17410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:04:54 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
Sep 27 22:04:54 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:04:54.059175 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.037754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.3.159.62:43719)
0927 22:04:54.038293 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:54.038298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:54.039209 (+ 911us) server_negotiation.cc:407] Connection header received
0927 22:04:54.039535 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:54.039545 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:54.039618 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:54.039744 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:54.041655 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.042774 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.045143 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.045678 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.045833 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.050686 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:54.050697 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:54.050703 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:54.050786 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:54.054790 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.056147 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.056157 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.056164 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.056363 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.056962 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.056966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.056968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.057344 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:54.057580 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:54.058324 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 22:04:54.058616 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":315,"threads_started":1}
I20260927 22:04:54.060518 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.028450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.3.159.60:38999)
0927 22:04:54.029739 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:54.029759 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:54.029830 (+ 71us) server_negotiation.cc:407] Connection header received
0927 22:04:54.032556 (+ 2726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:54.032733 (+ 177us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:54.033556 (+ 823us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:54.034152 (+ 596us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:54.037204 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.039805 (+ 2601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.042048 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.043359 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.043576 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.049630 (+ 6054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:54.049650 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:54.049670 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:54.049734 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:54.053794 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.055179 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.055190 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.055193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.055697 (+ 504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.056281 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.056286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.056288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.057247 (+ 959us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:54.057627 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:54.058130 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 22:04:54.058806 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":851,"spinlock_wait_cycles":37120,"thread_start_us":469,"threads_started":1}
I20260927 22:04:54.060580 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.037921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:43719 (local address 127.0.0.1:60768)
0927 22:04:54.038942 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:54.038998 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:54.039356 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:54.039834 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:54.039847 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:54.040929 (+ 1082us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:54.041338 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:54.041353 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.043075 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.043081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:54.044930 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:54.044948 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.045937 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.045944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:54.046118 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.047031 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:54.047082 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:54.050262 (+ 3180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:54.055155 (+ 4893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:54.055169 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:54.055188 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:54.055979 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:54.056513 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:54.056518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:54.056522 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:54.056740 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:54.057538 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:54.057548 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:54.057785 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 22:04:54.058137 (+ 352us) client_negotiation.cc:240] Negotiation successful
0927 22:04:54.059085 (+ 948us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":359,"thread_start_us":555,"threads_started":1}
I20260927 22:04:54.062986 17410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:04:54 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
Sep 27 22:04:54 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:04:54.085584 17335 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } attempt: 5
I20260927 22:04:54.086040 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.063932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:37583 (local address 127.0.0.1:57894)
0927 22:04:54.064217 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:54.064241 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:54.064413 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:54.066320 (+ 1907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:54.066329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:54.066940 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:54.067132 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:54.067142 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.068873 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.068880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:54.070448 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:54.070474 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.071762 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.071769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:54.072036 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.072972 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:54.073005 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:54.076901 (+ 3896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:54.081672 (+ 4771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:54.081678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:54.081682 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:54.082230 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:54.083293 (+ 1063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:54.083301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:54.083305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:54.083623 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:54.084649 (+ 1026us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:54.084655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:54.084879 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 22:04:54.085086 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 22:04:54.085614 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260927 22:04:54.087559 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.064112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.3.159.61:37583)
0927 22:04:54.065954 (+ 1842us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:54.065962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:54.065984 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:04:54.066050 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:54.066054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:54.066122 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:54.066271 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:54.067315 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.068632 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.070732 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.071526 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.071678 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.077248 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:54.077255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:54.077259 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:54.077308 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:54.081316 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.082591 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.082622 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.082625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.083041 (+ 416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.083891 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.083901 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.083906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.084488 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:54.084653 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:54.085299 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 22:04:54.085794 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1685,"thread_start_us":1541,"threads_started":1}
I20260927 22:04:54.087854 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.064973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.3.159.62:43719)
0927 22:04:54.065238 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:54.065245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:54.065271 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:04:54.065353 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:54.065358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:54.065561 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:54.065715 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:54.067459 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.068616 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.070732 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.071503 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.071679 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.078221 (+ 6542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:54.078234 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:54.078238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:54.078312 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:54.082221 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.083614 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.083622 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.083626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.083944 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.084423 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.084427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.084429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.084950 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:54.085307 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:54.087116 (+ 1809us) server_negotiation.cc:299] Negotiation successful
0927 22:04:54.087491 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 22:04:54.088999 17410 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:04:54 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
Sep 27 22:04:54 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546693, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
Sep 27 22:04:54 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
I20260927 22:04:54.108119 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.086414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51234)
0927 22:04:54.086657 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:54.086682 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:54.086809 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:54.087723 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:54.087729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:54.088416 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:54.088629 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:54.088640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.090146 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.090152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:54.091606 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:54.091631 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.093125 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.093133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:54.093464 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.094388 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:54.094421 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:54.097675 (+ 3254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:54.104475 (+ 6800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:54.104482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:54.104486 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:54.104997 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:54.105506 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:54.105512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:54.105516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:54.105708 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:54.106521 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:54.106526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:54.106755 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 22:04:54.106952 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 22:04:54.107451 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 22:04:54.108413 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.087177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.3.159.60:38999)
0927 22:04:54.087415 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:54.087421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:54.087438 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:04:54.087486 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:54.087490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:54.087554 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:54.087695 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:54.088799 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.089964 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.091844 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.092903 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.093084 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.098021 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:54.098030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:54.098035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:54.098092 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:54.104261 (+ 6169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.105141 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.105148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.105151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.105343 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.105860 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.105867 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.105871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.106336 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:54.106518 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:54.107659 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0927 22:04:54.107981 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 22:04:54.110069 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.090034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51240)
0927 22:04:54.091942 (+ 1908us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:54.091974 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:54.092108 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:54.092867 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:54.092871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:54.093588 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:54.093816 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:54.093833 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.095190 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.095195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:54.096755 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:54.096776 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.098111 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.098116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:54.098376 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.099163 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:54.099191 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:54.102454 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:54.106999 (+ 4545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:54.107007 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:54.107011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:54.107395 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:54.107815 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:54.107819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:54.107822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:54.107995 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:54.108872 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:54.108877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:54.109058 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 22:04:54.109245 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 22:04:54.109672 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1758}
I20260927 22:04:54.110563 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.090170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.3.159.60:38999)
0927 22:04:54.090975 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:54.090981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:54.092214 (+ 1233us) server_negotiation.cc:407] Connection header received
0927 22:04:54.092267 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:54.092271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:54.092339 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:54.092460 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:54.094042 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.094985 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.096977 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.097693 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.097833 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.102777 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:54.102783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:54.102787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:54.102821 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:54.106850 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.107518 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.107526 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.107531 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.107692 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.108116 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.108121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.108125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.108604 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:54.108920 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:54.109557 (+ 637us) server_negotiation.cc:299] Negotiation successful
0927 22:04:54.109952 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"spinlock_wait_cycles":27520,"thread_start_us":563,"threads_started":1}
I20260927 22:04:54.113762 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.090716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.3.159.61:37583)
0927 22:04:54.090983 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:54.090991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:54.091010 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:04:54.091078 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:54.091082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:54.091171 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:54.091292 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:54.093579 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.094746 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.097088 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.097884 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.098023 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.105775 (+ 7752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:54.105784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:54.105792 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:54.105831 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:54.109975 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.111100 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.111106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.111109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.111320 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.111879 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.111884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.111888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.112316 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:54.112665 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:54.113009 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 22:04:54.113319 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260927 22:04:54.117149 17335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb: Bootstrap starting.
I20260927 22:04:54.119920 17410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03: Bootstrap starting.
I20260927 22:04:54.121146 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:04:54.122422 17335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb: Neither blocks nor log segments found. Creating new log.
I20260927 22:04:54.123698 17410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03: Neither blocks nor log segments found. Creating new log.
I20260927 22:04:54.124119 17335 log.cc:824] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb: Log is configured to *not* fsync() on all Append() calls
I20260927 22:04:54.125255 17410 log.cc:824] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03: Log is configured to *not* fsync() on all Append() calls
I20260927 22:04:54.129148 17335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb: No bootstrap required, opened a new log
I20260927 22:04:54.134712 17410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03: No bootstrap required, opened a new log
I20260927 22:04:54.147802 17335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } }
I20260927 22:04:54.148240 17410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } }
I20260927 22:04:54.148729 17410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:04:54.148931 17410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f358d51483482cb9a2f78e1e21ca03, State: Initialized, Role: FOLLOWER
I20260927 22:04:54.149923 17261 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } attempt: 5
I20260927 22:04:54.150043 17410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } }
I20260927 22:04:54.153038 17335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:04:54.154541 17335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc119e7a1c948799b2516438e8c20fb, State: Initialized, Role: FOLLOWER
I20260927 22:04:54.156893 17335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } }
Sep 27 22:04:54 dist-test-slave-sg4j krb5kdc[17181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546692, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
I20260927 22:04:54.166026 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.129741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.3.159.60:38999)
0927 22:04:54.130005 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:54.130012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:54.136313 (+ 6301us) server_negotiation.cc:407] Connection header received
0927 22:04:54.136593 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:54.136599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:54.136704 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:54.136811 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:54.140917 (+ 4106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.142766 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.146787 (+ 4021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.147978 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.148159 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.154670 (+ 6511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:54.154680 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:54.154687 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:54.154750 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:54.160672 (+ 5922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.162605 (+ 1933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.162614 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.162620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.162903 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.163619 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.163627 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.163633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.164234 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:54.164842 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:54.165186 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 22:04:54.165542 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:04:54.172515 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.150759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38999 (local address 127.0.0.1:51250)
0927 22:04:54.151124 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 22:04:54.151153 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:04:54.151296 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:04:54.152092 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:04:54.152097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:04:54.152726 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:04:54.153011 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:54.153027 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.154569 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.154577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:54.156042 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:04:54.156062 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.157672 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.157681 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:04:54.157902 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.158699 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:04:54.158729 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:04:54.162126 (+ 3397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:04:54.168806 (+ 6680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:54.168812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:54.168816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:54.169364 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:54.169880 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:04:54.169884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:04:54.169888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:04:54.170142 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:04:54.171112 (+ 970us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:04:54.171121 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:04:54.171316 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 22:04:54.171467 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 22:04:54.171892 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"spinlock_wait_cycles":53120}
I20260927 22:04:54.174418 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.150844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.3.159.60:38999)
0927 22:04:54.151558 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:54.151564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:54.151586 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:04:54.151654 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:54.151659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:54.151753 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:54.151877 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:54.153209 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.154372 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.156293 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.157133 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.157290 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.162459 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:54.162471 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:54.162478 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:54.162534 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:54.167412 (+ 4878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.169556 (+ 2144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.169562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.169565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.169717 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.170286 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.170291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.170294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.170759 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:54.171000 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:54.173342 (+ 2342us) server_negotiation.cc:299] Negotiation successful
0927 22:04:54.173788 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563}
I20260927 22:04:54.172545 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.129521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.3.159.62:43719)
0927 22:04:54.129755 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:54.129760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:54.136077 (+ 6317us) server_negotiation.cc:407] Connection header received
0927 22:04:54.136333 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:54.136340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:54.136468 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:54.136652 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:54.148838 (+ 12186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.150177 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.153355 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.154096 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.154205 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.160914 (+ 6709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:54.160927 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:54.160932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:54.161001 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:54.165605 (+ 4604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.168650 (+ 3045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.168658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.168662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.168968 (+ 306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.169534 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.169539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.169542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.170011 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:54.170342 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:54.171315 (+ 973us) server_negotiation.cc:299] Negotiation successful
0927 22:04:54.171625 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"spinlock_wait_cycles":37632}
I20260927 22:04:54.180770 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } } }
I20260927 22:04:54.181670 17335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:04:54.182476 17261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8: Bootstrap starting.
I20260927 22:04:54.184208 17427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } } }
I20260927 22:04:54.184986 17427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:04:54.186056 17410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:04:54.181852 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:04:54.188726 17261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8: Neither blocks nor log segments found. Creating new log.
I20260927 22:04:54.191222 17261 log.cc:824] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:04:54.193441 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:54.136858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.3.159.61:37583)
0927 22:04:54.137066 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:54.137072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:54.154655 (+ 17583us) server_negotiation.cc:407] Connection header received
0927 22:04:54.154961 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:54.154967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:54.155079 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:54.155166 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:54.168490 (+ 13324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.170524 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.174171 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:54.175618 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:54.175804 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:54.181607 (+ 5803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:54.181619 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:54.181627 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:54.181709 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:54.186517 (+ 4808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.189116 (+ 2599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.189127 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.189135 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.189462 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:54.190197 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:54.190206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:54.190213 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:54.190801 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:54.191273 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:54.192302 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0927 22:04:54.192746 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:04:54.202797 17261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8: No bootstrap required, opened a new log
I20260927 22:04:54.220647 17261 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } }
I20260927 22:04:54.221275 17261 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:04:54.221473 17261 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9eda4de8694173b290ad70307810b8, State: Initialized, Role: FOLLOWER
I20260927 22:04:54.222522 17261 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } }
W20260927 22:04:54.222877 17451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:04:54.223265 17451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:04:54.226275 17454 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } } }
I20260927 22:04:54.226918 17454 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 22:04:54.227527 17449 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:04:54.227708 17449 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:04:54.228250 17261 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:04:54.218261 17343 master_runner.cc:186] Error getting master registration for 127.3.159.62:43719: OK, instance_id { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" instance_seqno: 1790546692682961 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:04:54.232986 17343 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:04:54.214498 17268 master_runner.cc:186] Error getting master registration for 127.3.159.62:43719: OK, instance_id { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" instance_seqno: 1790546692682961 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:04:54.235569 17268 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 22:04:54.238515 17364 tablet.cc:2369] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:04:54.256592 17194 master_runner.cc:263] No leader master found from master fc9eda4de8694173b290ad70307810b8
I20260927 22:04:54.256814 17194 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:04:54.257805 17454 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:04:54.258194 17454 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } }
I20260927 22:04:54.260352 17454 leader_election.cc:288] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fc119e7a1c948799b2516438e8c20fb (127.3.159.61:37583), 03f358d51483482cb9a2f78e1e21ca03 (127.3.159.60:38999)
I20260927 22:04:54.265008 17385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc9eda4de8694173b290ad70307810b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f358d51483482cb9a2f78e1e21ca03" is_pre_election: true
I20260927 22:04:54.265874 17385 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc9eda4de8694173b290ad70307810b8 in term 0.
I20260927 22:04:54.267452 17211 leader_election.cc:302] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f358d51483482cb9a2f78e1e21ca03, fc9eda4de8694173b290ad70307810b8; no voters:
I20260927 22:04:54.269229 17310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc9eda4de8694173b290ad70307810b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc119e7a1c948799b2516438e8c20fb" is_pre_election: true
I20260927 22:04:54.270056 17310 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc9eda4de8694173b290ad70307810b8 in term 0.
I20260927 22:04:54.272708 17454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:04:54.272934 17454 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:04:54.273056 17454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:04:54.277077 17454 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } }
I20260927 22:04:54.279157 17310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc9eda4de8694173b290ad70307810b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc119e7a1c948799b2516438e8c20fb"
I20260927 22:04:54.279484 17310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:04:54.289499 17310 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc9eda4de8694173b290ad70307810b8 in term 1.
I20260927 22:04:54.292487 17213 leader_election.cc:302] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc119e7a1c948799b2516438e8c20fb, fc9eda4de8694173b290ad70307810b8; no voters:
I20260927 22:04:54.296427 17385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fc9eda4de8694173b290ad70307810b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f358d51483482cb9a2f78e1e21ca03"
I20260927 22:04:54.296759 17385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:04:54.300712 17467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:04:54.300890 17385 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc9eda4de8694173b290ad70307810b8 in term 1.
I20260927 22:04:54.302052 17454 leader_election.cc:288] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [CANDIDATE]: Term 1 election: Requested vote from peers 4fc119e7a1c948799b2516438e8c20fb (127.3.159.61:37583), 03f358d51483482cb9a2f78e1e21ca03 (127.3.159.60:38999)
W20260927 22:04:54.310709 17289 tablet.cc:2369] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:04:54.311092 17465 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:04:54.311301 17465 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:04:54.311801 17467 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [term 1 LEADER]: Becoming Leader. State: Replica: fc9eda4de8694173b290ad70307810b8, State: Running, Role: LEADER
I20260927 22:04:54.312610 17467 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } }
I20260927 22:04:54.330956 17454 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc9eda4de8694173b290ad70307810b8. Latest consensus state: current_term: 1 leader_uuid: "fc9eda4de8694173b290ad70307810b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } } }
I20260927 22:04:54.331382 17454 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:04:54.338450 17470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:04:54.341845 17470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:04:54.377204 17385 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [term 1 FOLLOWER]: Refusing update from remote peer fc9eda4de8694173b290ad70307810b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:04:54.378695 17467 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:04:54.390607 17310 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [term 1 FOLLOWER]: Refusing update from remote peer fc9eda4de8694173b290ad70307810b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:04:54.392549 17467 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:04:54.416318 17215 tablet.cc:2369] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:04:54.437992 17427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc9eda4de8694173b290ad70307810b8. Latest consensus state: current_term: 1 leader_uuid: "fc9eda4de8694173b290ad70307810b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } } }
I20260927 22:04:54.438439 17427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:04:54.441884 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc9eda4de8694173b290ad70307810b8. Latest consensus state: current_term: 1 leader_uuid: "fc9eda4de8694173b290ad70307810b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } } }
I20260927 22:04:54.442227 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:04:54.444875 17472 mvcc.cc:205] Tried to move back new op lower bound from 7334079260153753600 to 7334079259911749632. Current Snapshot: MvccSnapshot[applied={T|T < 7334079260153753600}]
I20260927 22:04:54.448173 17473 mvcc.cc:205] Tried to move back new op lower bound from 7334079260153753600 to 7334079259911749632. Current Snapshot: MvccSnapshot[applied={T|T < 7334079260153753600}]
I20260927 22:04:54.455981 17427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc9eda4de8694173b290ad70307810b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } } }
I20260927 22:04:54.456377 17427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:04:54.458602 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc9eda4de8694173b290ad70307810b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } } }
I20260927 22:04:54.458953 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:04:54.471989 17468 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc9eda4de8694173b290ad70307810b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } } }
I20260927 22:04:54.472625 17468 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:04:54.472996 17468 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc9eda4de8694173b290ad70307810b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc9eda4de8694173b290ad70307810b8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 43719 } } peers { permanent_uuid: "4fc119e7a1c948799b2516438e8c20fb" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 37583 } } peers { permanent_uuid: "03f358d51483482cb9a2f78e1e21ca03" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38999 } } }
I20260927 22:04:54.473284 17468 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:04:54.473806 17470 catalog_manager.cc:1347] Generated new cluster ID: 8dbc124f00a349e2b3a9895331bf558b
I20260927 22:04:54.473929 17470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:04:54.507025 17470 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:04:54.509331 17470 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:04:54.552268 17470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc9eda4de8694173b290ad70307810b8: Generated new TSK 0
I20260927 22:04:54.553640 17470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:04:55.226271 17451 catalog_manager.cc:1259] Loaded cluster ID: 8dbc124f00a349e2b3a9895331bf558b
I20260927 22:04:55.226451 17451 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03: loading cluster ID for follower catalog manager: success
I20260927 22:04:55.229806 17451 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03: acquiring CA information for follower catalog manager: success
I20260927 22:04:55.230726 17449 catalog_manager.cc:1259] Loaded cluster ID: 8dbc124f00a349e2b3a9895331bf558b
I20260927 22:04:55.230834 17449 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb: loading cluster ID for follower catalog manager: success
I20260927 22:04:55.232524 17451 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03f358d51483482cb9a2f78e1e21ca03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:04:55.234723 17449 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb: acquiring CA information for follower catalog manager: success
I20260927 22:04:55.237077 17449 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fc119e7a1c948799b2516438e8c20fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:04:55.730166 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 17194
I20260927 22:04:55.732455 17256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:55.860255 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 17194
I20260927 22:04:55.890267 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 17268
I20260927 22:04:55.891707 17330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:56.016185 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 17268
I20260927 22:04:56.043210 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 17343
I20260927 22:04:56.044600 17405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:56.161223 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 17343
2026-09-27T22:04:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5495 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:04:56 dist-test-slave-sg4j krb5kdc[17488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:04:56 dist-test-slave-sg4j krb5kdc[17488](info): set up 2 sockets
Sep 27 22:04:56 dist-test-slave-sg4j krb5kdc[17488](info): commencing operation
krb5kdc: starting...
W20260927 22:04:57.380745 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:04:57 dist-test-slave-sg4j krb5kdc[17488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:04:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:04:57.707552 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:33331
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:33331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:04:58.164757 17501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:04:58.165270 17501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:04:58.165475 17501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:04:58.174441 17501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:04:58.174695 17501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:04:58.174814 17501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:04:58.174923 17501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:04:58.175019 17501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:04:58.187961 17501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:33331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:33331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:04:58.189980 17501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:04:58.191915 17501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:04:58.211009 17509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:58.211141 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:04:58.211091 17507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:04:58.212823 17501 server_base.cc:1034] running on GCE node
I20260927 22:04:58.214162 17501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:04:58.216475 17501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:04:58.218014 17501 hybrid_clock.cc:648] HybridClock initialized: now 1790546698217923 us; error 78 us; skew 500 ppm
I20260927 22:04:58.218643 17501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:04:58 dist-test-slave-sg4j krb5kdc[17488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546698, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:04:58.223697 17501 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:04:58.226243 17501 webserver.cc:460] Webserver started at http://127.3.159.62:33545/ using document root <none> and password file <none>
I20260927 22:04:58.227308 17501 fs_manager.cc:359] Metadata directory not provided
I20260927 22:04:58.227504 17501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:04:58.227934 17501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:04:58.232548 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "e009f369ee9c47568b9e68214cf377d6"
format_stamp: "Formatted at 2026-09-27 22:04:58 on dist-test-slave-sg4j"
server_key: "06eac86a26f56eeee2c7441f317fc92c"
server_key_iv: "8954254a16ef02d110bab95b04b8b507"
server_key_version: "encryptionkey@0"
I20260927 22:04:58.234030 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "e009f369ee9c47568b9e68214cf377d6"
format_stamp: "Formatted at 2026-09-27 22:04:58 on dist-test-slave-sg4j"
server_key: "06eac86a26f56eeee2c7441f317fc92c"
server_key_iv: "8954254a16ef02d110bab95b04b8b507"
server_key_version: "encryptionkey@0"
I20260927 22:04:58.242061 17501 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 22:04:58.247659 17516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:04:58.250159 17501 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:04:58.250512 17501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "e009f369ee9c47568b9e68214cf377d6"
format_stamp: "Formatted at 2026-09-27 22:04:58 on dist-test-slave-sg4j"
server_key: "06eac86a26f56eeee2c7441f317fc92c"
server_key_iv: "8954254a16ef02d110bab95b04b8b507"
server_key_version: "encryptionkey@0"
I20260927 22:04:58.250846 17501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:04:58.275672 17501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:04:58.282477 17501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:04:58.282998 17501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:04:58.309080 17501 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:33331
I20260927 22:04:58.309139 17567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:33331 every 8 connection(s)
I20260927 22:04:58.311110 17501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:04:58.313104 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 17501
I20260927 22:04:58.313441 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:04:58.314498 3708 external_mini_cluster.cc:1428] Setting key 2cc0e2400cdf44c4c8ed6e351b55e306
I20260927 22:04:58.317512 17568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:04:58.333312 17568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6: Bootstrap starting.
Sep 27 22:04:58 dist-test-slave-sg4j krb5kdc[17488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:04:58.338652 17568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6: Neither blocks nor log segments found. Creating new log.
I20260927 22:04:58.340003 17568 log.cc:824] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:04:58.345050 17568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6: No bootstrap required, opened a new log
I20260927 22:04:58.351344 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:04:58.317204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.3.159.62:33331)
0927 22:04:58.318480 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0927 22:04:58.318496 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:04:58.318556 (+ 60us) server_negotiation.cc:407] Connection header received
0927 22:04:58.320687 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:04:58.320729 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:04:58.321373 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:04:58.321983 (+ 610us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:04:58.324193 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:58.326063 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:58.328233 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:04:58.329212 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:04:58.329536 (+ 324us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:04:58.339323 (+ 9787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:04:58.339336 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:04:58.339361 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:04:58.339440 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:04:58.344547 (+ 5107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:58.346018 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:58.346028 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:58.346038 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:58.346322 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:04:58.347055 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:04:58.347060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:04:58.347062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:04:58.347927 (+ 865us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:04:58.348294 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 22:04:58.348941 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 22:04:58.349700 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":594,"threads_started":1}
I20260927 22:04:58.356858 17568 raft_consensus.cc:348] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e009f369ee9c47568b9e68214cf377d6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33331 } }
I20260927 22:04:58.357596 17568 raft_consensus.cc:374] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:04:58.357792 17568 raft_consensus.cc:729] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e009f369ee9c47568b9e68214cf377d6, State: Initialized, Role: FOLLOWER
I20260927 22:04:58.358392 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:04:58.358851 17568 consensus_queue.cc:260] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e009f369ee9c47568b9e68214cf377d6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33331 } }
I20260927 22:04:58.359484 17568 raft_consensus.cc:388] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:04:58.359692 17568 raft_consensus.cc:482] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:04:58.359906 17568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:04:58.362843 17568 raft_consensus.cc:504] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e009f369ee9c47568b9e68214cf377d6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33331 } }
I20260927 22:04:58.363575 17568 leader_election.cc:302] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e009f369ee9c47568b9e68214cf377d6; no voters:
I20260927 22:04:58.364265 17568 leader_election.cc:288] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:04:58.364557 17573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:04:58.365622 17573 raft_consensus.cc:686] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [term 1 LEADER]: Becoming Leader. State: Replica: e009f369ee9c47568b9e68214cf377d6, State: Running, Role: LEADER
I20260927 22:04:58.366470 17573 consensus_queue.cc:237] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e009f369ee9c47568b9e68214cf377d6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33331 } }
I20260927 22:04:58.366986 17568 sys_catalog.cc:564] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:04:58.370883 17574 sys_catalog.cc:455] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e009f369ee9c47568b9e68214cf377d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e009f369ee9c47568b9e68214cf377d6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33331 } } }
I20260927 22:04:58.371551 17574 sys_catalog.cc:458] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:04:58.370859 17575 sys_catalog.cc:455] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e009f369ee9c47568b9e68214cf377d6. Latest consensus state: current_term: 1 leader_uuid: "e009f369ee9c47568b9e68214cf377d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e009f369ee9c47568b9e68214cf377d6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 33331 } } }
I20260927 22:04:58.371697 17575 sys_catalog.cc:458] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:04:58.372778 17582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:04:58.384667 17582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:04:58.403100 17582 catalog_manager.cc:1347] Generated new cluster ID: 4f1f9ad99f1f4ab1b0f5d5d3e97bc1ae
I20260927 22:04:58.403472 17582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:04:58.421105 17582 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:04:58.422734 17582 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:04:58.441751 17582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e009f369ee9c47568b9e68214cf377d6: Generated new TSK 0
I20260927 22:04:58.443459 17582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:04:58.476362 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 17501
I20260927 22:04:58.478250 17563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:04:58.588021 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 17501
2026-09-27T22:04:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2423 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:04:58 dist-test-slave-sg4j krb5kdc[17596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:04:58 dist-test-slave-sg4j krb5kdc[17596](info): set up 2 sockets
Sep 27 22:04:58 dist-test-slave-sg4j krb5kdc[17596](info): commencing operation
krb5kdc: starting...
W20260927 22:04:59.802416 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.004s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:04:59 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:05:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:05:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:05:00.108250 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37253
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:05:00.586303 17609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:00.586768 17609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:00.586879 17609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:00.596144 17609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:05:00.596362 17609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:00.596424 17609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:00.596483 17609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:05:00.596526 17609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:05:00.608608 17609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:37253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:00.611135 17609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:00.613306 17609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:00.632012 17614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:00.632105 17615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:00.632196 17617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:00.633283 17609 server_base.cc:1034] running on GCE node
I20260927 22:05:00.634406 17609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:00.637029 17609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:00.638505 17609 hybrid_clock.cc:648] HybridClock initialized: now 1790546700638475 us; error 145 us; skew 500 ppm
I20260927 22:05:00.639810 17609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:00 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546700, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:00.644928 17609 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:05:00.647260 17609 webserver.cc:460] Webserver started at http://127.3.159.62:43471/ using document root <none> and password file <none>
I20260927 22:05:00.648458 17609 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:00.648718 17609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:00.649154 17609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:05:00.653740 17609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "4d54427316934863a1174cf933f9fd46"
format_stamp: "Formatted at 2026-09-27 22:05:00 on dist-test-slave-sg4j"
server_key: "0ad165fcfc68f17edebc71cc9a0eb9f8"
server_key_iv: "dbdb96b54963dbe1fa82f044bed0d2f6"
server_key_version: "encryptionkey@0"
I20260927 22:05:00.655442 17609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "4d54427316934863a1174cf933f9fd46"
format_stamp: "Formatted at 2026-09-27 22:05:00 on dist-test-slave-sg4j"
server_key: "0ad165fcfc68f17edebc71cc9a0eb9f8"
server_key_iv: "dbdb96b54963dbe1fa82f044bed0d2f6"
server_key_version: "encryptionkey@0"
I20260927 22:05:00.663708 17609 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260927 22:05:00.669911 17624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:00.672271 17609 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:05:00.672621 17609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "4d54427316934863a1174cf933f9fd46"
format_stamp: "Formatted at 2026-09-27 22:05:00 on dist-test-slave-sg4j"
server_key: "0ad165fcfc68f17edebc71cc9a0eb9f8"
server_key_iv: "dbdb96b54963dbe1fa82f044bed0d2f6"
server_key_version: "encryptionkey@0"
I20260927 22:05:00.672951 17609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:00.697731 17609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:00.704263 17609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:00.704679 17609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:00.730731 17609 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:37253
I20260927 22:05:00.730772 17675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:37253 every 8 connection(s)
I20260927 22:05:00.732721 17609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:05:00.738345 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 17609
I20260927 22:05:00.738641 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:05:00.739369 17676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:00.739910 3708 external_mini_cluster.cc:1428] Setting key 20fb4fd6d642db54f4965be6b02493d2
I20260927 22:05:00.756155 17676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Bootstrap starting.
Sep 27 22:05:00 dist-test-slave-sg4j krb5kdc[17596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:00.760409 17676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:00.762531 17676 log.cc:824] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:00.767295 17676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: No bootstrap required, opened a new log
I20260927 22:05:00.773505 17679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:00.743014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42552 (local address 127.3.159.62:37253)
0927 22:05:00.744939 (+ 1925us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:00.744961 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:00.745037 (+ 76us) server_negotiation.cc:407] Connection header received
0927 22:05:00.748402 (+ 3365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:00.748451 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:00.749206 (+ 755us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:00.749897 (+ 691us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:00.751763 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:00.753460 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:00.755024 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:00.755946 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:00.756213 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:00.762050 (+ 5837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:00.762063 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:00.762085 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:00.762149 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:00.767247 (+ 5098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:00.768460 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:00.768472 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:00.768483 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:00.768776 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:00.769324 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:00.769329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:00.769331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:00.770264 (+ 933us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:00.770764 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:00.771277 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 22:05:00.771921 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1543,"thread_start_us":880,"threads_started":1}
I20260927 22:05:00.777287 17676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:00.777997 17676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:00.778149 17676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d54427316934863a1174cf933f9fd46, State: Initialized, Role: FOLLOWER
I20260927 22:05:00.779762 17676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:00.780198 17676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:05:00.780364 17676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:05:00.780593 17676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:00.785125 17676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:00.786165 17676 leader_election.cc:302] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d54427316934863a1174cf933f9fd46; no voters:
I20260927 22:05:00.787547 17676 leader_election.cc:288] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:05:00.787716 17681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:05:00.788942 17681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 1 LEADER]: Becoming Leader. State: Replica: 4d54427316934863a1174cf933f9fd46, State: Running, Role: LEADER
I20260927 22:05:00.789822 17681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:00.791049 17676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:05:00.794401 17683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d54427316934863a1174cf933f9fd46. Latest consensus state: current_term: 1 leader_uuid: "4d54427316934863a1174cf933f9fd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } } }
I20260927 22:05:00.795082 17683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: This master's current role is: LEADER
I20260927 22:05:00.795909 17682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d54427316934863a1174cf933f9fd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } } }
I20260927 22:05:00.796420 17682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: This master's current role is: LEADER
I20260927 22:05:00.798655 17689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:05:00.808848 17689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:05:00.821764 17689 catalog_manager.cc:1347] Generated new cluster ID: 9a474ea08ee94886bd32d423564d4fed
I20260927 22:05:00.821949 17689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:05:00.841944 17689 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:05:00.844578 17689 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:05:00.856994 17689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Generated new TSK 0
I20260927 22:05:00.858350 17689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:05:00.956188 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37253
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:05:01.503783 17702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:01.504359 17702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:01.504714 17702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:01.514371 17702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:01.514534 17702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:01.514673 17702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:05:01.528286 17702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:01.529980 17702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:01.531464 17702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:01.550900 17707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:01.551373 17708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:01.552052 17710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:01.552413 17702 server_base.cc:1034] running on GCE node
I20260927 22:05:01.553676 17702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:01.555133 17702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:01.556594 17702 hybrid_clock.cc:648] HybridClock initialized: now 1790546701556496 us; error 87 us; skew 500 ppm
I20260927 22:05:01.557092 17702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:01 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:01.561923 17702 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:05:01.564671 17702 webserver.cc:460] Webserver started at http://127.3.159.1:45643/ using document root <none> and password file <none>
I20260927 22:05:01.566053 17702 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:01.566529 17702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:01.566972 17702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:05:01.571796 17702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "8a39b17a706f4a46923a6f80563307c2"
format_stamp: "Formatted at 2026-09-27 22:05:01 on dist-test-slave-sg4j"
server_key: "9deef8e4ecd289aa1b61da00f18b3556"
server_key_iv: "b5f08114eb7b14f3775ea7900d8b1081"
server_key_version: "encryptionkey@0"
I20260927 22:05:01.573679 17702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "8a39b17a706f4a46923a6f80563307c2"
format_stamp: "Formatted at 2026-09-27 22:05:01 on dist-test-slave-sg4j"
server_key: "9deef8e4ecd289aa1b61da00f18b3556"
server_key_iv: "b5f08114eb7b14f3775ea7900d8b1081"
server_key_version: "encryptionkey@0"
I20260927 22:05:01.581037 17702 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260927 22:05:01.586484 17717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:01.588552 17702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:05:01.588866 17702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "8a39b17a706f4a46923a6f80563307c2"
format_stamp: "Formatted at 2026-09-27 22:05:01 on dist-test-slave-sg4j"
server_key: "9deef8e4ecd289aa1b61da00f18b3556"
server_key_iv: "b5f08114eb7b14f3775ea7900d8b1081"
server_key_version: "encryptionkey@0"
I20260927 22:05:01.589138 17702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:01.608002 17702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:01.613473 17702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:01.613994 17702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:01.615334 17702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:01.617188 17702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:01.617311 17702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:01.617525 17702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:01.617621 17702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:01.662173 17702 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:45755
I20260927 22:05:01.662251 17829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:45755 every 8 connection(s)
I20260927 22:05:01.663797 17702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:05:01.665903 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 17702
I20260927 22:05:01.666177 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:05:01.667145 3708 external_mini_cluster.cc:1428] Setting key b7c4d2cec6f8a380314bf02adba11f7c
Sep 27 22:05:01 dist-test-slave-sg4j krb5kdc[17596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546701, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:01.696028 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:01.667670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:48353 (local address 127.3.159.62:37253)
0927 22:05:01.668716 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:01.668725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:01.669829 (+ 1104us) server_negotiation.cc:407] Connection header received
0927 22:05:01.672172 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:01.672181 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:01.672299 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:01.672471 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:01.674942 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:01.676331 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:01.678894 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:01.679546 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:01.679842 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:01.685728 (+ 5886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:01.685736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:01.685741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:01.685796 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:01.689561 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:01.691347 (+ 1786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:01.691354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:01.691357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:01.691603 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:01.692701 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:01.692711 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:01.692716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:01.693171 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:01.693519 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:01.695331 (+ 1812us) server_negotiation.cc:299] Negotiation successful
0927 22:05:01.695623 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":856,"thread_start_us":698,"threads_started":1}
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:05:01.698097 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:01.668430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:48353)
0927 22:05:01.669612 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:01.669681 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:01.671766 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:01.672829 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:01.672841 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:01.673727 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:01.674583 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:01.674610 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:01.676661 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:01.676667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:01.678526 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:01.678549 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:01.680008 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:01.680019 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:01.680393 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:01.681475 (+ 1082us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:01.681525 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:01.685085 (+ 3560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:01.689862 (+ 4777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:01.689881 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:01.689915 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:01.691024 (+ 1109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:01.691785 (+ 761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:01.691794 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:01.691802 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:01.692236 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:01.693494 (+ 1258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:01.693511 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:01.694397 (+ 886us) client_negotiation.cc:769] Sending connection context
0927 22:05:01.694904 (+ 507us) client_negotiation.cc:240] Negotiation successful
0927 22:05:01.695942 (+ 1038us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":480,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:05:01.700918 17830 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37253
I20260927 22:05:01.701517 17830 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:01.702587 17830 heartbeater.cc:511] Master 127.3.159.62:37253 requested a full tablet report, sending...
I20260927 22:05:01.705963 17641 ts_manager.cc:194] Registered new tserver with Master: 8a39b17a706f4a46923a6f80563307c2 (127.3.159.1:45755)
I20260927 22:05:01.708623 17641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:48353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:05:01.742612 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37253
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:05:02.241874 17836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:02.242277 17836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:02.242450 17836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:02.251905 17836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:02.252130 17836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:02.252358 17836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:05:02.263984 17836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:02.265606 17836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:02.267374 17836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:02.284911 17842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:02.284976 17841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:02.285663 17844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:02.285996 17836 server_base.cc:1034] running on GCE node
I20260927 22:05:02.286847 17836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:02.288291 17836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:02.289652 17836 hybrid_clock.cc:648] HybridClock initialized: now 1790546702289587 us; error 71 us; skew 500 ppm
I20260927 22:05:02.290132 17836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:02 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546702, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:02.294922 17836 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:05:02.297292 17836 webserver.cc:460] Webserver started at http://127.3.159.2:36765/ using document root <none> and password file <none>
I20260927 22:05:02.298475 17836 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:02.298616 17836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:02.299327 17836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:05:02.302989 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "fb62c320be2e46b4b756588d7271e80d"
format_stamp: "Formatted at 2026-09-27 22:05:02 on dist-test-slave-sg4j"
server_key: "662592e5f71b9d7c422052410a6b0b21"
server_key_iv: "a1fbfffc2aeecd96e2a2346041a062ef"
server_key_version: "encryptionkey@0"
I20260927 22:05:02.303790 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "fb62c320be2e46b4b756588d7271e80d"
format_stamp: "Formatted at 2026-09-27 22:05:02 on dist-test-slave-sg4j"
server_key: "662592e5f71b9d7c422052410a6b0b21"
server_key_iv: "a1fbfffc2aeecd96e2a2346041a062ef"
server_key_version: "encryptionkey@0"
I20260927 22:05:02.310554 17836 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260927 22:05:02.315091 17851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:02.317169 17836 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260927 22:05:02.317473 17836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "fb62c320be2e46b4b756588d7271e80d"
format_stamp: "Formatted at 2026-09-27 22:05:02 on dist-test-slave-sg4j"
server_key: "662592e5f71b9d7c422052410a6b0b21"
server_key_iv: "a1fbfffc2aeecd96e2a2346041a062ef"
server_key_version: "encryptionkey@0"
I20260927 22:05:02.317757 17836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:02.339449 17836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:02.345629 17836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:02.345968 17836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:02.347366 17836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:02.349128 17836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:02.349223 17836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:02.349392 17836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:02.349486 17836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 22:05:02.399417 17836 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:35481
I20260927 22:05:02.399582 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:35481 every 8 connection(s)
I20260927 22:05:02.401701 17836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:05:02.403833 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 17836
I20260927 22:05:02.404075 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:05:02.404562 3708 external_mini_cluster.cc:1428] Setting key 4c0fb8cfdd31b756680a786b2041210b
Sep 27 22:05:02 dist-test-slave-sg4j krb5kdc[17596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546702, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
I20260927 22:05:02.433660 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:02.405843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:43075 (local address 127.3.159.62:37253)
0927 22:05:02.406771 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:02.406779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:02.408611 (+ 1832us) server_negotiation.cc:407] Connection header received
0927 22:05:02.411264 (+ 2653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:02.411271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:02.411399 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:02.411546 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:02.414123 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:02.414993 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:02.417099 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:02.417696 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:02.417869 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:02.423480 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:02.423488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:02.423492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:02.423560 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:02.427294 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:02.429124 (+ 1830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:02.429130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:02.429133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:02.429456 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:02.430456 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:02.430461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:02.430463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:02.430849 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:02.431009 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:02.432891 (+ 1882us) server_negotiation.cc:299] Negotiation successful
0927 22:05:02.433194 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":413,"threads_started":1}
I20260927 22:05:02.434715 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:02.406807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:43075)
0927 22:05:02.408355 (+ 1548us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:02.408420 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:02.410775 (+ 2355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:02.411961 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:02.411974 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:02.412787 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:02.413811 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:02.413843 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:02.415210 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:02.415215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:02.416865 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:02.416884 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:02.418005 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:02.418013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:02.418331 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:02.419215 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:02.419246 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:02.422951 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:02.427620 (+ 4669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:02.427638 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:02.427674 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:02.428807 (+ 1133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:02.429754 (+ 947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:02.429764 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:02.429769 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:02.430239 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:02.431066 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:02.431081 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:02.431830 (+ 749us) client_negotiation.cc:769] Sending connection context
0927 22:05:02.432677 (+ 847us) client_negotiation.cc:240] Negotiation successful
0927 22:05:02.433304 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"thread_start_us":622,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
I20260927 22:05:02.437268 17964 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37253
I20260927 22:05:02.437817 17964 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:02.439011 17964 heartbeater.cc:511] Master 127.3.159.62:37253 requested a full tablet report, sending...
I20260927 22:05:02.442129 17641 ts_manager.cc:194] Registered new tserver with Master: fb62c320be2e46b4b756588d7271e80d (127.3.159.2:35481)
I20260927 22:05:02.443352 17641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:43075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:05:02.474606 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37253
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:05:02.714046 17830 heartbeater.cc:503] Master 127.3.159.62:37253 was elected leader, sending a full tablet report...
W20260927 22:05:02.891486 17970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:02.891999 17970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:02.892202 17970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:02.901160 17970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:02.901326 17970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:02.901558 17970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:05:02.916059 17970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:02.918103 17970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:02.919862 17970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:02.938868 17976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:02.938750 17975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:02.938843 17978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:02.940137 17970 server_base.cc:1034] running on GCE node
I20260927 22:05:02.941100 17970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:02.942626 17970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:02.943979 17970 hybrid_clock.cc:648] HybridClock initialized: now 1790546702943920 us; error 53 us; skew 500 ppm
I20260927 22:05:02.944510 17970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:02 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546702, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:02.950278 17970 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:05:02.952919 17970 webserver.cc:460] Webserver started at http://127.3.159.3:45143/ using document root <none> and password file <none>
I20260927 22:05:02.954334 17970 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:02.954530 17970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:02.955065 17970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:05:02.959226 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "51bd0c16c7d14937b2b981894ad3a43f"
format_stamp: "Formatted at 2026-09-27 22:05:02 on dist-test-slave-sg4j"
server_key: "c35ce838b601a1e7ffba3944f7b5c99d"
server_key_iv: "b7f3417563b7e1188586280c8637b352"
server_key_version: "encryptionkey@0"
I20260927 22:05:02.960299 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "51bd0c16c7d14937b2b981894ad3a43f"
format_stamp: "Formatted at 2026-09-27 22:05:02 on dist-test-slave-sg4j"
server_key: "c35ce838b601a1e7ffba3944f7b5c99d"
server_key_iv: "b7f3417563b7e1188586280c8637b352"
server_key_version: "encryptionkey@0"
I20260927 22:05:02.968140 17970 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 22:05:02.973621 17985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:02.976140 17970 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:05:02.976438 17970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "51bd0c16c7d14937b2b981894ad3a43f"
format_stamp: "Formatted at 2026-09-27 22:05:02 on dist-test-slave-sg4j"
server_key: "c35ce838b601a1e7ffba3944f7b5c99d"
server_key_iv: "b7f3417563b7e1188586280c8637b352"
server_key_version: "encryptionkey@0"
I20260927 22:05:02.976753 17970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:02.999087 17970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:03.005301 17970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:03.005735 17970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:03.007146 17970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:03.009084 17970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:03.009178 17970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:03.009338 17970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:03.009425 17970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 22:05:03.056484 17970 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:40403
I20260927 22:05:03.056562 18097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:40403 every 8 connection(s)
I20260927 22:05:03.058391 17970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:05:03.067909 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 17970
I20260927 22:05:03.068159 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:05:03.068723 3708 external_mini_cluster.cc:1428] Setting key e976c2129c2b8bcdd590136edd9fe3b7
Sep 27 22:05:03 dist-test-slave-sg4j krb5kdc[17596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546702, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:03.086500 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.062055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:50093 (local address 127.3.159.62:37253)
0927 22:05:03.062649 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:03.062655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:03.064204 (+ 1549us) server_negotiation.cc:407] Connection header received
0927 22:05:03.066664 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:03.066672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:03.066787 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:03.066951 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:03.068953 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.070069 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.072213 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.072738 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.072915 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.077738 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:03.077745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:03.077748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:03.077801 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:03.081668 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:03.082879 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:03.082886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:03.082897 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:03.083136 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:03.083945 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:03.083953 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:03.083956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:03.084458 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:03.084720 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:03.085771 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0927 22:05:03.086064 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":318,"threads_started":1}
I20260927 22:05:03.087543 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.062768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:50093)
0927 22:05:03.063918 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:03.063985 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:03.066261 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:03.067253 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:03.067263 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:03.067933 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:03.068742 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.068763 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.070268 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.070273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.072020 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.072036 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.073035 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.073044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.073334 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.074275 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:03.074307 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:03.077321 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:03.081954 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:03.081970 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:03.081999 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:03.082713 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:03.083281 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:03.083286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:03.083290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:03.083687 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:03.084675 (+ 988us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:03.084687 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:03.085239 (+ 552us) client_negotiation.cc:769] Sending connection context
0927 22:05:03.085643 (+ 404us) client_negotiation.cc:240] Negotiation successful
0927 22:05:03.086273 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":453,"threads_started":1}
I20260927 22:05:03.089305 18098 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37253
I20260927 22:05:03.089704 18098 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:03.090574 18098 heartbeater.cc:511] Master 127.3.159.62:37253 requested a full tablet report, sending...
I20260927 22:05:03.092712 17641 ts_manager.cc:194] Registered new tserver with Master: 51bd0c16c7d14937b2b981894ad3a43f (127.3.159.3:40403)
I20260927 22:05:03.093983 17641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:50093
I20260927 22:05:03.100828 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:05:03.125667 18100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.107037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42554 (local address 127.3.159.62:37253)
0927 22:05:03.107459 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:03.107464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:03.107829 (+ 365us) server_negotiation.cc:407] Connection header received
0927 22:05:03.107983 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:03.107989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:03.108104 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:03.108282 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:03.110201 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.111201 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.112947 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.113522 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.113679 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.117069 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:03.117077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:03.117081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:03.117139 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:03.121284 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:03.122613 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:03.122618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:03.122620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:03.122831 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:03.123412 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:03.123415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:03.123418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:03.123830 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:03.124307 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:03.124759 (+ 452us) server_negotiation.cc:299] Negotiation successful
0927 22:05:03.125177 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260927 22:05:03.131600 17641 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:05:03.140573 17641 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:05:03.174588 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.160112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:40403 (local address 127.0.0.1:52930)
0927 22:05:03.161499 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:03.161547 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:03.161790 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:03.164060 (+ 2270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:03.164070 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:03.164112 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:03.164852 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.164875 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.167636 (+ 2761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.167644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.170424 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.170451 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.173086 (+ 2635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.173096 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.173546 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.173634 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 22:05:03.173818 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 22:05:03.174097 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1030,"thread_start_us":862,"threads_started":1}
I20260927 22:05:03.176635 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.160417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.3.159.3:40403)
0927 22:05:03.163411 (+ 2994us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:03.163419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:03.163453 (+ 34us) server_negotiation.cc:407] Connection header received
0927 22:05:03.163550 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:03.163556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:03.163748 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:03.165148 (+ 1400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:03.165552 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.167398 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.170733 (+ 3335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.172690 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.175430 (+ 2740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.175823 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:03.175933 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 22:05:03.176189 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2798,"thread_start_us":2592,"threads_started":1}
I20260927 22:05:03.181352 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.161169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.3.159.1:45755)
0927 22:05:03.162664 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:03.162675 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:03.163155 (+ 480us) server_negotiation.cc:407] Connection header received
0927 22:05:03.163317 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:03.163328 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:03.163751 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:03.164009 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:03.166648 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.168513 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.173307 (+ 4794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.178089 (+ 4782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.178328 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.178664 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:03.180026 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0927 22:05:03.180311 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1239,"thread_start_us":1003,"threads_started":1}
I20260927 22:05:03.183312 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.160479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:45755 (local address 127.0.0.1:56728)
0927 22:05:03.162803 (+ 2324us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:03.162831 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:03.162997 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:03.164332 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:03.164339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:03.164363 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:03.164938 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.164951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.168740 (+ 3789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.168748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.172995 (+ 4247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.173015 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.179340 (+ 6325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.179349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.179654 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.179710 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:05:03.179814 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 22:05:03.182735 (+ 2921us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2137,"mutex_wait_us":1,"thread_start_us":1992,"threads_started":1}
I20260927 22:05:03.185671 18033 tablet_service.cc:1467] Processing CreateTablet for tablet f0e7668e89304dcbb5e9d76e89250078 (DEFAULT_TABLE table=test-table [id=8f110dcae82b4dc9b3ab7d529185fdf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:03.187867 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.163265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:35481 (local address 127.0.0.1:45478)
0927 22:05:03.174980 (+ 11715us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:03.175005 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:03.175164 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:03.179295 (+ 4131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:03.179308 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:03.179354 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:03.179669 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.179682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.182660 (+ 2978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.182667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.184449 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.184483 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.186774 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.186784 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.187059 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.187119 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 22:05:03.187298 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 22:05:03.187495 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11594,"thread_start_us":16009,"threads_started":1}
I20260927 22:05:03.188316 18033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0e7668e89304dcbb5e9d76e89250078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:03.189684 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.163923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.3.159.2:35481)
0927 22:05:03.178188 (+ 14265us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:03.178202 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:03.178246 (+ 44us) server_negotiation.cc:407] Connection header received
0927 22:05:03.178419 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:03.178430 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:03.178921 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:03.179926 (+ 1005us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:03.180249 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.182275 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.184715 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.186476 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.186664 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.186880 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:03.188832 (+ 1952us) server_negotiation.cc:299] Negotiation successful
0927 22:05:03.189088 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14020,"thread_start_us":13027,"threads_started":1}
I20260927 22:05:03.189982 17765 tablet_service.cc:1467] Processing CreateTablet for tablet f0e7668e89304dcbb5e9d76e89250078 (DEFAULT_TABLE table=test-table [id=8f110dcae82b4dc9b3ab7d529185fdf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:03.191707 17765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0e7668e89304dcbb5e9d76e89250078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:03.194178 17899 tablet_service.cc:1467] Processing CreateTablet for tablet f0e7668e89304dcbb5e9d76e89250078 (DEFAULT_TABLE table=test-table [id=8f110dcae82b4dc9b3ab7d529185fdf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:03.196302 17899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0e7668e89304dcbb5e9d76e89250078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:03.209046 18117 tablet_bootstrap.cc:492] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: Bootstrap starting.
I20260927 22:05:03.214462 18118 tablet_bootstrap.cc:492] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2: Bootstrap starting.
I20260927 22:05:03.215075 18117 tablet_bootstrap.cc:654] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:03.217173 18117 log.cc:824] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:03.220222 18118 tablet_bootstrap.cc:654] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:03.222211 18120 tablet_bootstrap.cc:492] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d: Bootstrap starting.
I20260927 22:05:03.222523 18118 log.cc:824] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:03.226656 18120 tablet_bootstrap.cc:654] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:03.226792 18118 tablet_bootstrap.cc:492] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2: No bootstrap required, opened a new log
I20260927 22:05:03.227239 18118 ts_tablet_manager.cc:1397] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260927 22:05:03.227962 18120 log.cc:824] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:03.230118 18117 tablet_bootstrap.cc:492] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: No bootstrap required, opened a new log
I20260927 22:05:03.230530 18117 ts_tablet_manager.cc:1397] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.010s
I20260927 22:05:03.231323 18120 tablet_bootstrap.cc:492] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d: No bootstrap required, opened a new log
I20260927 22:05:03.231655 18120 ts_tablet_manager.cc:1397] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260927 22:05:03.240096 18120 raft_consensus.cc:348] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } }
I20260927 22:05:03.240588 18120 raft_consensus.cc:374] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:03.240705 18120 raft_consensus.cc:729] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb62c320be2e46b4b756588d7271e80d, State: Initialized, Role: FOLLOWER
I20260927 22:05:03.240489 18118 raft_consensus.cc:348] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } }
I20260927 22:05:03.241132 18118 raft_consensus.cc:374] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:03.241245 18118 raft_consensus.cc:729] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a39b17a706f4a46923a6f80563307c2, State: Initialized, Role: FOLLOWER
I20260927 22:05:03.241509 18120 consensus_queue.cc:260] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } }
I20260927 22:05:03.242195 18118 consensus_queue.cc:260] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } }
I20260927 22:05:03.243258 17964 heartbeater.cc:503] Master 127.3.159.62:37253 was elected leader, sending a full tablet report...
I20260927 22:05:03.244452 18120 ts_tablet_manager.cc:1428] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 22:05:03.244220 18117 raft_consensus.cc:348] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } }
I20260927 22:05:03.245052 18117 raft_consensus.cc:374] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:03.245208 18117 raft_consensus.cc:729] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bd0c16c7d14937b2b981894ad3a43f, State: Initialized, Role: FOLLOWER
I20260927 22:05:03.246343 18117 consensus_queue.cc:260] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } }
I20260927 22:05:03.248108 18098 heartbeater.cc:503] Master 127.3.159.62:37253 was elected leader, sending a full tablet report...
I20260927 22:05:03.249431 18117 ts_tablet_manager.cc:1428] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: Time spent starting tablet: real 0.019s user 0.018s sys 0.000s
I20260927 22:05:03.253597 18118 ts_tablet_manager.cc:1428] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2: Time spent starting tablet: real 0.026s user 0.023s sys 0.001s
I20260927 22:05:03.275110 18125 raft_consensus.cc:482] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:05:03.275511 18125 raft_consensus.cc:504] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } }
I20260927 22:05:03.277961 18125 leader_election.cc:288] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a39b17a706f4a46923a6f80563307c2 (127.3.159.1:45755), fb62c320be2e46b4b756588d7271e80d (127.3.159.2:35481)
I20260927 22:05:03.288995 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.278807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:43029 (local address 127.3.159.1:45755)
0927 22:05:03.279095 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:03.279104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:03.279299 (+ 195us) server_negotiation.cc:407] Connection header received
0927 22:05:03.279666 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:03.279672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:03.279789 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:03.279892 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:03.280649 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.281831 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.285695 (+ 3864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.287344 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.287586 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.287728 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:03.288448 (+ 720us) server_negotiation.cc:299] Negotiation successful
0927 22:05:03.288651 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260927 22:05:03.289042 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.279779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:35049 (local address 127.3.159.2:35481)
0927 22:05:03.280002 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:03.280011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:03.280595 (+ 584us) server_negotiation.cc:407] Connection header received
0927 22:05:03.280697 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:03.280706 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:03.280804 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:03.280979 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:03.281673 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.282601 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.285731 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.287335 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.287586 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.287728 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:03.288448 (+ 720us) server_negotiation.cc:299] Negotiation successful
0927 22:05:03.288651 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 22:05:03.289075 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.278704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:45755 (local address 127.3.159.3:43029)
0927 22:05:03.279092 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:03.279120 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:03.279414 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:03.280137 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:03.280143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:03.280180 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:03.280445 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.280465 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.282032 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.282037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.285236 (+ 3199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.285272 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.287743 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.287752 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.288086 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.288157 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 22:05:03.288316 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 22:05:03.288584 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":178}
I20260927 22:05:03.289144 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.279387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:35481 (local address 127.3.159.3:35049)
0927 22:05:03.280403 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:03.280427 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:03.280578 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:03.281027 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:03.281031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:03.281056 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:03.281501 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.281512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.283007 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.283014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.285502 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:03.285516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.287780 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.287792 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:03.288114 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.288159 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 22:05:03.288331 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 22:05:03.288581 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":138,"thread_start_us":631,"threads_started":1}
I20260927 22:05:03.291343 17919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e7668e89304dcbb5e9d76e89250078" candidate_uuid: "51bd0c16c7d14937b2b981894ad3a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb62c320be2e46b4b756588d7271e80d" is_pre_election: true
I20260927 22:05:03.291327 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e7668e89304dcbb5e9d76e89250078" candidate_uuid: "51bd0c16c7d14937b2b981894ad3a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a39b17a706f4a46923a6f80563307c2" is_pre_election: true
I20260927 22:05:03.292511 17919 raft_consensus.cc:2454] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bd0c16c7d14937b2b981894ad3a43f in term 0.
I20260927 22:05:03.292586 17785 raft_consensus.cc:2454] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bd0c16c7d14937b2b981894ad3a43f in term 0.
I20260927 22:05:03.293802 17986 leader_election.cc:302] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bd0c16c7d14937b2b981894ad3a43f, fb62c320be2e46b4b756588d7271e80d; no voters:
I20260927 22:05:03.294409 18125 raft_consensus.cc:2781] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:05:03.294647 18125 raft_consensus.cc:482] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:05:03.294828 18125 raft_consensus.cc:3037] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:03.298237 18125 raft_consensus.cc:504] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } }
I20260927 22:05:03.299283 18125 leader_election.cc:288] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [CANDIDATE]: Term 1 election: Requested vote from peers 8a39b17a706f4a46923a6f80563307c2 (127.3.159.1:45755), fb62c320be2e46b4b756588d7271e80d (127.3.159.2:35481)
I20260927 22:05:03.300099 17785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e7668e89304dcbb5e9d76e89250078" candidate_uuid: "51bd0c16c7d14937b2b981894ad3a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a39b17a706f4a46923a6f80563307c2"
I20260927 22:05:03.300189 17919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0e7668e89304dcbb5e9d76e89250078" candidate_uuid: "51bd0c16c7d14937b2b981894ad3a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb62c320be2e46b4b756588d7271e80d"
I20260927 22:05:03.300426 17785 raft_consensus.cc:3037] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:03.300483 17919 raft_consensus.cc:3037] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:03.303213 17785 raft_consensus.cc:2454] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bd0c16c7d14937b2b981894ad3a43f in term 1.
I20260927 22:05:03.303581 17919 raft_consensus.cc:2454] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51bd0c16c7d14937b2b981894ad3a43f in term 1.
I20260927 22:05:03.303877 17986 leader_election.cc:302] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bd0c16c7d14937b2b981894ad3a43f, 8a39b17a706f4a46923a6f80563307c2; no voters:
I20260927 22:05:03.304397 18125 raft_consensus.cc:2781] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:05:03.305007 18125 raft_consensus.cc:686] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 1 LEADER]: Becoming Leader. State: Replica: 51bd0c16c7d14937b2b981894ad3a43f, State: Running, Role: LEADER
I20260927 22:05:03.305756 18125 consensus_queue.cc:237] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } }
W20260927 22:05:03.310595 18099 tablet.cc:2369] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:05:03.311967 17640 catalog_manager.cc:5495] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f reported cstate change: term changed from 0 to 1, leader changed from <none> to 51bd0c16c7d14937b2b981894ad3a43f (127.3.159.3). New cstate: current_term: 1 leader_uuid: "51bd0c16c7d14937b2b981894ad3a43f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } health_report { overall_health: UNKNOWN } } }
I20260927 22:05:03.345475 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.339454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.3.159.3:40403)
0927 22:05:03.339751 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:03.339757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:03.339784 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:05:03.339900 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:03.339905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:03.340016 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:03.340237 (+ 221us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:05:03.340731 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.341588 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.343099 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.343526 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.343668 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.344249 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:05:03.344807 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:05:03.344987 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:03.345048 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 22:05:03.345137 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260927 22:05:03.357935 17919 raft_consensus.cc:1261] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d [term 1 FOLLOWER]: Refusing update from remote peer 51bd0c16c7d14937b2b981894ad3a43f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:05:03.358247 17785 raft_consensus.cc:1261] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2 [term 1 FOLLOWER]: Refusing update from remote peer 51bd0c16c7d14937b2b981894ad3a43f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:05:03.359877 18127 consensus_queue.cc:1035] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:05:03.360674 18125 consensus_queue.cc:1035] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:05:03.398396 17640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:05:03.399261 17640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:05:03.410090 17640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Sending DeleteTablet for 3 replicas of tablet f0e7668e89304dcbb5e9d76e89250078
I20260927 22:05:03.411809 17765 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e7668e89304dcbb5e9d76e89250078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:03 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:56728
I20260927 22:05:03.412321 18033 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e7668e89304dcbb5e9d76e89250078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:03 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:52930
I20260927 22:05:03.419334 17899 tablet_service.cc:1514] Processing DeleteTablet for tablet f0e7668e89304dcbb5e9d76e89250078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:03 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:45478
I20260927 22:05:03.423874 18141 tablet_replica.cc:330] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: stopping tablet replica
I20260927 22:05:03.423758 18140 tablet_replica.cc:330] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2: stopping tablet replica
I20260927 22:05:03.424304 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 17702
I20260927 22:05:03.425046 18142 tablet_replica.cc:330] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d: stopping tablet replica
I20260927 22:05:03.425091 18140 raft_consensus.cc:2229] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:05:03.425935 18140 raft_consensus.cc:2258] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:03.426220 18142 raft_consensus.cc:2229] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:05:03.427100 18142 raft_consensus.cc:2258] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:03.427026 18141 raft_consensus.cc:2229] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:05:03.428428 18141 raft_consensus.cc:2258] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:03.430306 18142 ts_tablet_manager.cc:1905] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:03.435667 18140 ts_tablet_manager.cc:1905] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:03.440999 18142 ts_tablet_manager.cc:1918] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:03.441497 18142 log.cc:1196] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/f0e7668e89304dcbb5e9d76e89250078
I20260927 22:05:03.442875 18142 ts_tablet_manager.cc:1939] T f0e7668e89304dcbb5e9d76e89250078 P fb62c320be2e46b4b756588d7271e80d: Deleting consensus metadata
I20260927 22:05:03.444811 18141 ts_tablet_manager.cc:1905] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:03.447786 17625 catalog_manager.cc:4841] TS fb62c320be2e46b4b756588d7271e80d (127.3.159.2:35481): tablet f0e7668e89304dcbb5e9d76e89250078 (table test-table [id=8f110dcae82b4dc9b3ab7d529185fdf0]) successfully deleted
I20260927 22:05:03.449041 18140 ts_tablet_manager.cc:1918] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:03.450579 18140 log.cc:1196] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/f0e7668e89304dcbb5e9d76e89250078
I20260927 22:05:03.451272 18140 ts_tablet_manager.cc:1939] T f0e7668e89304dcbb5e9d76e89250078 P 8a39b17a706f4a46923a6f80563307c2: Deleting consensus metadata
I20260927 22:05:03.453771 17625 catalog_manager.cc:4841] TS 8a39b17a706f4a46923a6f80563307c2 (127.3.159.1:45755): tablet f0e7668e89304dcbb5e9d76e89250078 (table test-table [id=8f110dcae82b4dc9b3ab7d529185fdf0]) successfully deleted
I20260927 22:05:03.454397 18141 ts_tablet_manager.cc:1918] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:03.454677 18141 log.cc:1196] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/f0e7668e89304dcbb5e9d76e89250078
I20260927 22:05:03.455461 18141 ts_tablet_manager.cc:1939] T f0e7668e89304dcbb5e9d76e89250078 P 51bd0c16c7d14937b2b981894ad3a43f: Deleting consensus metadata
I20260927 22:05:03.457751 17626 catalog_manager.cc:4841] TS 51bd0c16c7d14937b2b981894ad3a43f (127.3.159.3:40403): tablet f0e7668e89304dcbb5e9d76e89250078 (table test-table [id=8f110dcae82b4dc9b3ab7d529185fdf0]) successfully deleted
Sep 27 22:05:03 dist-test-slave-sg4j krb5kdc[17596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:05:03.473944 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.426127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.3.159.1:45755)
0927 22:05:03.426333 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:03.426339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:03.441333 (+ 14994us) server_negotiation.cc:407] Connection header received
0927 22:05:03.442346 (+ 1013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:03.442353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:03.442483 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:03.442646 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:03.448621 (+ 5975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.450165 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.452870 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.453929 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.454096 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.465590 (+ 11494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:03.465616 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:03.465640 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:03.465707 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:03.469592 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:03.470752 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:03.470763 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:03.470766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:03.471172 (+ 406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:03.471814 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:03.471818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:03.471821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:03.472370 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:03.472746 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:03.473160 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 22:05:03.473495 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:05:03.474757 17825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:03.581897 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 17702
I20260927 22:05:03.606237 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 17836
Sep 27 22:05:03 dist-test-slave-sg4j krb5kdc[17596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:05:03.628804 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.607239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.3.159.2:35481)
0927 22:05:03.607543 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:03.607552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:03.607582 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:05:03.607669 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:03.607677 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:03.607795 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:03.607943 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:03.609457 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.610646 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.612305 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.613374 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.613843 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.619697 (+ 5854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:03.619722 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:03.619747 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:03.619809 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:03.623751 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:03.624972 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:03.624982 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:03.624985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:03.625152 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:03.626055 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:03.626071 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:03.626074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:03.627065 (+ 991us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:03.627413 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:03.628009 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 22:05:03.628294 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 22:05:03.629901 17959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:03.751475 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 17836
I20260927 22:05:03.775663 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 17970
Sep 27 22:05:03 dist-test-slave-sg4j krb5kdc[17596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:05:03.800688 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:03.777554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.3.159.3:40403)
0927 22:05:03.777902 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:03.777922 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:03.777965 (+ 43us) server_negotiation.cc:407] Connection header received
0927 22:05:03.778085 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:03.778094 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:03.778205 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:03.778411 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:03.780959 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.782461 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.784882 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:03.785578 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:03.785740 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:03.792902 (+ 7162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:03.792927 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:03.792949 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:03.793016 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:03.796823 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:03.797988 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:03.797997 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:03.798000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:03.798210 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:03.798642 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:03.798646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:03.798648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:03.799272 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:03.799665 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:03.799964 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 22:05:03.800277 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 22:05:03.801616 18093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:03.945044 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 17970
I20260927 22:05:03.969954 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 17609
I20260927 22:05:03.971642 17671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:04.090641 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 17609
I20260927 22:05:04.120702 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37253
--webserver_interface=127.3.159.62
--webserver_port=43471
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:05:04.546041 18148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:04.546415 18148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:04.546599 18148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:04.556988 18148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:05:04.557192 18148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:04.557302 18148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:04.557436 18148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:05:04.557515 18148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:05:04.571708 18148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:37253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=43471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:04.573366 18148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:04.574990 18148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:04.605453 18154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:04.605746 18155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:04.605541 18157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:04.608814 18148 server_base.cc:1034] running on GCE node
I20260927 22:05:04.610340 18148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:04.612696 18148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:04.613978 18148 hybrid_clock.cc:648] HybridClock initialized: now 1790546704613887 us; error 75 us; skew 500 ppm
I20260927 22:05:04.614564 18148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:04 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546704, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:04.619441 18148 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:05:04.622210 18148 webserver.cc:460] Webserver started at http://127.3.159.62:43471/ using document root <none> and password file <none>
I20260927 22:05:04.623649 18148 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:04.623875 18148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:04.632493 18148 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:05:04.637316 18164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:04.639685 18148 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260927 22:05:04.640010 18148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "4d54427316934863a1174cf933f9fd46"
format_stamp: "Formatted at 2026-09-27 22:05:00 on dist-test-slave-sg4j"
server_key: "0ad165fcfc68f17edebc71cc9a0eb9f8"
server_key_iv: "dbdb96b54963dbe1fa82f044bed0d2f6"
server_key_version: "encryptionkey@0"
I20260927 22:05:04.641610 18148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:04.658529 18148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:04.665153 18148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:04.665585 18148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:04.692641 18148 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:37253
I20260927 22:05:04.692704 18215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:37253 every 8 connection(s)
I20260927 22:05:04.694844 18148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:05:04.703665 18216 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:05:04.705001 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 18148
I20260927 22:05:04.706046 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:45755
--local_ip_for_outbound_sockets=127.3.159.1
--tserver_master_addrs=127.3.159.62:37253
--webserver_port=45643
--webserver_interface=127.3.159.1
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:05:04.708088 18216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Bootstrap starting.
I20260927 22:05:04.729955 18216 log.cc:824] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:04.748322 18216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:05:04.749089 18216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Bootstrap complete.
I20260927 22:05:04.761044 18216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:04.762228 18216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d54427316934863a1174cf933f9fd46, State: Initialized, Role: FOLLOWER
I20260927 22:05:04.763192 18216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:04.763500 18216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:05:04.763638 18216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:05:04.763873 18216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:05:04.769780 18216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:04.770678 18216 leader_election.cc:302] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d54427316934863a1174cf933f9fd46; no voters:
I20260927 22:05:04.771539 18216 leader_election.cc:288] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:05:04.771832 18220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:05:04.775069 18216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:05:04.778200 18220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 2 LEADER]: Becoming Leader. State: Replica: 4d54427316934863a1174cf933f9fd46, State: Running, Role: LEADER
I20260927 22:05:04.779176 18220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:04.783795 18221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4d54427316934863a1174cf933f9fd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } } }
I20260927 22:05:04.784346 18221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: This master's current role is: LEADER
I20260927 22:05:04.785099 18222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d54427316934863a1174cf933f9fd46. Latest consensus state: current_term: 2 leader_uuid: "4d54427316934863a1174cf933f9fd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } } }
I20260927 22:05:04.785672 18222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: This master's current role is: LEADER
I20260927 22:05:04.795004 18230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:05:04.807224 18230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:05:04.808950 18230 catalog_manager.cc:1259] Loaded cluster ID: 9a474ea08ee94886bd32d423564d4fed
I20260927 22:05:04.809214 18230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:05:04.812999 18230 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 22:05:04.834379 18230 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 22:05:04.848661 18230 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:05:04.851336 18230 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Loaded TSK: 0
I20260927 22:05:04.853528 18230 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:05:05.196835 18218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:05.197525 18218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:05.198366 18218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:05.212795 18218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:05.213017 18218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:05.213234 18218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:05:05.227061 18218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:45755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=45643
--tserver_master_addrs=127.3.159.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:05.229524 18218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:05.232075 18218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:05.254842 18245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:05.254894 18247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:05.256029 18244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:05.256059 18218 server_base.cc:1034] running on GCE node
I20260927 22:05:05.257104 18218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:05.258688 18218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:05.260118 18218 hybrid_clock.cc:648] HybridClock initialized: now 1790546705260014 us; error 94 us; skew 500 ppm
I20260927 22:05:05.260568 18218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:05 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:05.265185 18218 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:05:05.268185 18218 webserver.cc:460] Webserver started at http://127.3.159.1:45643/ using document root <none> and password file <none>
I20260927 22:05:05.269510 18218 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:05.269901 18218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:05.278736 18218 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 22:05:05.283900 18254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:05.287000 18218 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260927 22:05:05.287374 18218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "8a39b17a706f4a46923a6f80563307c2"
format_stamp: "Formatted at 2026-09-27 22:05:01 on dist-test-slave-sg4j"
server_key: "9deef8e4ecd289aa1b61da00f18b3556"
server_key_iv: "b5f08114eb7b14f3775ea7900d8b1081"
server_key_version: "encryptionkey@0"
I20260927 22:05:05.289029 18218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:05.319383 18218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:05.326038 18218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:05.326471 18218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:05.328148 18218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:05.330338 18218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:05.330536 18218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:05.330711 18218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:05.330826 18218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:05.384037 18218 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:45755
I20260927 22:05:05.384171 18366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:45755 every 8 connection(s)
I20260927 22:05:05.385949 18218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:05:05.393707 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 18218
I20260927 22:05:05.394589 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:35481
--local_ip_for_outbound_sockets=127.3.159.2
--tserver_master_addrs=127.3.159.62:37253
--webserver_port=36765
--webserver_interface=127.3.159.2
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:05:05 dist-test-slave-sg4j krb5kdc[17596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:05.421653 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:05.391181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:60595)
0927 22:05:05.392492 (+ 1311us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:05.392557 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:05.395382 (+ 2825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:05.397829 (+ 2447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:05.397844 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:05.398885 (+ 1041us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:05.399697 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:05.399725 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:05.401863 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:05.401868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:05.403559 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:05.403585 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:05.405058 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:05.405068 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:05.405637 (+ 569us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:05.406722 (+ 1085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:05.406756 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:05.410311 (+ 3555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:05.415407 (+ 5096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:05.415425 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:05.415453 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:05.416107 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:05.416732 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:05.416738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:05.416743 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:05.416983 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:05.418157 (+ 1174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:05.418168 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:05.418642 (+ 474us) client_negotiation.cc:769] Sending connection context
0927 22:05:05.419001 (+ 359us) client_negotiation.cc:240] Negotiation successful
0927 22:05:05.419814 (+ 813us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":579,"threads_started":1}
I20260927 22:05:05.421597 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:05.390776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:60595 (local address 127.3.159.62:37253)
0927 22:05:05.391594 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:05.391613 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:05.392805 (+ 1192us) server_negotiation.cc:407] Connection header received
0927 22:05:05.396378 (+ 3573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:05.396395 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:05.396979 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:05.397606 (+ 627us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:05.400052 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:05.401583 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:05.403822 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:05.404684 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:05.404994 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:05.410902 (+ 5908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:05.410916 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:05.410940 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:05.411007 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:05.415104 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:05.416325 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:05.416335 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:05.416341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:05.416567 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:05.417171 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:05.417186 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:05.417188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:05.417961 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:05.418235 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:05.419286 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0927 22:05:05.419821 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":338,"threads_started":1}
I20260927 22:05:05.426883 18367 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37253
I20260927 22:05:05.427376 18367 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:05.428378 18367 heartbeater.cc:511] Master 127.3.159.62:37253 requested a full tablet report, sending...
I20260927 22:05:05.431660 18181 ts_manager.cc:194] Registered new tserver with Master: 8a39b17a706f4a46923a6f80563307c2 (127.3.159.1:45755)
I20260927 22:05:05.434000 18181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:60595
W20260927 22:05:05.788919 18371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:05.789376 18371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:05.789583 18371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:05.798122 18371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:05.798336 18371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:05.798532 18371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:05:05.810851 18371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:35481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=36765
--tserver_master_addrs=127.3.159.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:05.812704 18371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:05.814440 18371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:05.831135 18377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:05.831310 18376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:05.831279 18379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:05.832512 18371 server_base.cc:1034] running on GCE node
I20260927 22:05:05.833556 18371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:05.835101 18371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:05.836503 18371 hybrid_clock.cc:648] HybridClock initialized: now 1790546705836427 us; error 65 us; skew 500 ppm
I20260927 22:05:05.837167 18371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:05 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:05.842082 18371 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:05:05.844807 18371 webserver.cc:460] Webserver started at http://127.3.159.2:36765/ using document root <none> and password file <none>
I20260927 22:05:05.846028 18371 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:05.846196 18371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:05.854475 18371 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 22:05:05.858909 18386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:05.861132 18371 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:05:05.861500 18371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "fb62c320be2e46b4b756588d7271e80d"
format_stamp: "Formatted at 2026-09-27 22:05:02 on dist-test-slave-sg4j"
server_key: "662592e5f71b9d7c422052410a6b0b21"
server_key_iv: "a1fbfffc2aeecd96e2a2346041a062ef"
server_key_version: "encryptionkey@0"
I20260927 22:05:05.862798 18371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:05.888281 18371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:05.894193 18371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:05.894598 18371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:05.896132 18371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:05.898134 18371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:05.898247 18371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:05.898346 18371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:05.898423 18371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:05.947945 18371 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:35481
I20260927 22:05:05.948053 18498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:35481 every 8 connection(s)
I20260927 22:05:05.949985 18371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:05:05.957044 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 18371
I20260927 22:05:05.957827 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:40403
--local_ip_for_outbound_sockets=127.3.159.3
--tserver_master_addrs=127.3.159.62:37253
--webserver_port=45143
--webserver_interface=127.3.159.3
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:05:05 dist-test-slave-sg4j krb5kdc[17596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546705, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:05.977654 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:05.953348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:33881 (local address 127.3.159.62:37253)
0927 22:05:05.954087 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:05.954095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:05.955417 (+ 1322us) server_negotiation.cc:407] Connection header received
0927 22:05:05.958323 (+ 2906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:05.958334 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:05.958484 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:05.958664 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:05.961002 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:05.962673 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:05.964913 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:05.965494 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:05.965655 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:05.970337 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:05.970343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:05.970347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:05.970396 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:05.973635 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:05.974601 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:05.974607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:05.974611 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:05.974772 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:05.975295 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:05.975302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:05.975305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:05.975657 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:05.975825 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:05.976837 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0927 22:05:05.977133 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":342,"threads_started":1}
I20260927 22:05:05.978981 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:05.954062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:33881)
0927 22:05:05.955156 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:05.955216 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:05.957853 (+ 2637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:05.959037 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:05.959049 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:05.959792 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:05.960705 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:05.960728 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:05.963030 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:05.963036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:05.964623 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:05.964641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:05.965826 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:05.965835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:05.966129 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:05.967077 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:05.967112 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:05.970030 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:05.973834 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:05.973846 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:05.973884 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:05.974448 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:05.974920 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:05.974924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:05.974930 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:05.975176 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:05.975805 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:05.975815 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:05.976386 (+ 571us) client_negotiation.cc:769] Sending connection context
0927 22:05:05.976743 (+ 357us) client_negotiation.cc:240] Negotiation successful
0927 22:05:05.977502 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":405,"threads_started":1}
I20260927 22:05:05.981258 18499 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37253
I20260927 22:05:05.981694 18499 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:05.982719 18499 heartbeater.cc:511] Master 127.3.159.62:37253 requested a full tablet report, sending...
I20260927 22:05:05.984656 18181 ts_manager.cc:194] Registered new tserver with Master: fb62c320be2e46b4b756588d7271e80d (127.3.159.2:35481)
I20260927 22:05:05.985672 18181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:33881
W20260927 22:05:06.349293 18503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:06.349699 18503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:06.349879 18503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:06.359373 18503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:06.359920 18503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:06.360288 18503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:05:06.373310 18503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:40403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=45143
--tserver_master_addrs=127.3.159.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:06.375043 18503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:06.376667 18503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:06.395668 18509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:06.396791 18508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:06.397356 18503 server_base.cc:1034] running on GCE node
W20260927 22:05:06.397362 18511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:06.398612 18503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:06.399976 18503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:06.401314 18503 hybrid_clock.cc:648] HybridClock initialized: now 1790546706401223 us; error 108 us; skew 500 ppm
I20260927 22:05:06.401820 18503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:06 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546706, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:06.406674 18503 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:05:06.409327 18503 webserver.cc:460] Webserver started at http://127.3.159.3:45143/ using document root <none> and password file <none>
I20260927 22:05:06.410338 18503 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:06.410521 18503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:06.418008 18503 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 22:05:06.422400 18518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:06.425446 18503 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260927 22:05:06.425870 18503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "51bd0c16c7d14937b2b981894ad3a43f"
format_stamp: "Formatted at 2026-09-27 22:05:02 on dist-test-slave-sg4j"
server_key: "c35ce838b601a1e7ffba3944f7b5c99d"
server_key_iv: "b7f3417563b7e1188586280c8637b352"
server_key_version: "encryptionkey@0"
I20260927 22:05:06.427197 18503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:06.438695 18367 heartbeater.cc:503] Master 127.3.159.62:37253 was elected leader, sending a full tablet report...
I20260927 22:05:06.458964 18503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:06.464109 18503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:06.464394 18503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:06.465637 18503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:06.467366 18503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:06.467478 18503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:06.467593 18503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:06.467684 18503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:06.510149 18503 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:40403
I20260927 22:05:06.510346 18630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:40403 every 8 connection(s)
I20260927 22:05:06.511770 18503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:05:06.518694 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 18503
Sep 27 22:05:06 dist-test-slave-sg4j krb5kdc[17596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546706, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:06.542644 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.520280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.3.159.62:37253)
0927 22:05:06.521170 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:06.521177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:06.521931 (+ 754us) server_negotiation.cc:407] Connection header received
0927 22:05:06.522372 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:06.522376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:06.522444 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:06.522558 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:06.524908 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.525990 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.528158 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.528769 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.528945 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.533370 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:06.533423 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:06.533429 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:06.533486 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:06.537802 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:06.539178 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:06.539185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:06.539189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:06.539665 (+ 476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:06.540388 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:06.540392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:06.540395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:06.540874 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:06.541170 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:06.541894 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 22:05:06.542216 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":560,"threads_started":1}
I20260927 22:05:06.543818 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.515375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:54427 (local address 127.3.159.62:37253)
0927 22:05:06.515998 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:06.516003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:06.517072 (+ 1069us) server_negotiation.cc:407] Connection header received
0927 22:05:06.520170 (+ 3098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:06.520177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:06.520293 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:06.520445 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:06.523321 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.524222 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.526856 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.527342 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.527483 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.534040 (+ 6557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:06.534047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:06.534050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:06.534133 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:06.538506 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:06.540147 (+ 1641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:06.540152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:06.540156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:06.540364 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:06.541025 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:06.541030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:06.541033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:06.541471 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:06.541624 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:06.543004 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0927 22:05:06.543336 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":358,"threads_started":1}
I20260927 22:05:06.545117 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.515816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:54427)
0927 22:05:06.516848 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:06.516904 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:06.519716 (+ 2812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:06.520828 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:06.520847 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:06.521926 (+ 1079us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:06.523001 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.523039 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.524514 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.524520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.526588 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.526612 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.527675 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.527684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.528093 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.529272 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:06.529317 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:06.533561 (+ 4244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:06.538768 (+ 5207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:06.538792 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:06.538816 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:06.539962 (+ 1146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:06.540520 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:06.540528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:06.540533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:06.540836 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:06.541667 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:06.541677 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:06.542358 (+ 681us) client_negotiation.cc:769] Sending connection context
0927 22:05:06.542753 (+ 395us) client_negotiation.cc:240] Negotiation successful
0927 22:05:06.543447 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":416,"threads_started":1}
I20260927 22:05:06.547230 18631 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37253
I20260927 22:05:06.547751 18631 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:06.548789 18631 heartbeater.cc:511] Master 127.3.159.62:37253 requested a full tablet report, sending...
I20260927 22:05:06.551082 18180 ts_manager.cc:194] Registered new tserver with Master: 51bd0c16c7d14937b2b981894ad3a43f (127.3.159.3:40403)
I20260927 22:05:06.552395 18180 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:54427
I20260927 22:05:06.561251 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:05:06.571352 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.562622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.3.159.62:37253)
0927 22:05:06.562873 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:06.562878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:06.563400 (+ 522us) server_negotiation.cc:407] Connection header received
0927 22:05:06.563760 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:06.563768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:06.564004 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:06.564197 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:05:06.565030 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.566012 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.568064 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.568749 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.568892 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.569723 (+ 831us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:05:06.570527 (+ 804us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:05:06.570702 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:06.570906 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 22:05:06.571016 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 22:05:06.572940 18180 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:05:06.583034 18180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:05:06.617262 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.600196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.3.159.2:35481)
0927 22:05:06.600950 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:06.600962 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:06.602279 (+ 1317us) server_negotiation.cc:407] Connection header received
0927 22:05:06.603203 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:06.603217 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:06.603615 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:06.603838 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:06.606362 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.608555 (+ 2193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.613706 (+ 5151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.615461 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.615567 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.615794 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:06.616651 (+ 857us) server_negotiation.cc:299] Negotiation successful
0927 22:05:06.616860 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":423,"threads_started":1}
I20260927 22:05:06.618338 18642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.599941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:35481 (local address 127.0.0.1:56016)
0927 22:05:06.602085 (+ 2144us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:06.602139 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:06.602458 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:06.605358 (+ 2900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:06.605369 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:06.605442 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:06.606070 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.606089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.609853 (+ 3764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.609862 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.613365 (+ 3503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.613418 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.616039 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.616049 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.616362 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.616430 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 22:05:06.617561 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0927 22:05:06.617805 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1546,"spinlock_wait_cycles":58240,"thread_start_us":1832,"threads_started":1}
I20260927 22:05:06.624301 18434 tablet_service.cc:1467] Processing CreateTablet for tablet cda78f1feaa14d79a8a2a9b39f59e2f9 (DEFAULT_TABLE table=test-table [id=b3306aee226841d8ac7014f3d9ca742b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:06.625500 18644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.600974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:40403 (local address 127.0.0.1:34558)
0927 22:05:06.602072 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:06.602195 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:06.602460 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:06.615820 (+ 13360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:06.615828 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:06.615872 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:06.616361 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.616374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.618739 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.618745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.622195 (+ 3450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.622213 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.624466 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.624475 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.624666 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.624716 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 22:05:06.624797 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 22:05:06.624991 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":609,"threads_started":1}
I20260927 22:05:06.626184 18646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.602840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:45755 (local address 127.0.0.1:43922)
0927 22:05:06.615727 (+ 12887us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:06.615759 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:06.615995 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:06.617873 (+ 1878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:06.617878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:06.617906 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:06.618383 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.618394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.620105 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.620114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.623235 (+ 3121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.623256 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.624982 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.624990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.625247 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.625291 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:05:06.625541 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 22:05:06.625731 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12653,"thread_start_us":13334,"threads_started":1}
I20260927 22:05:06.626472 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.601509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.3.159.3:40403)
0927 22:05:06.614778 (+ 13269us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:06.614790 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:06.614830 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:05:06.614955 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:06.614964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:06.615273 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:06.616602 (+ 1329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:06.616911 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.618576 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.622470 (+ 3894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.624137 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.625351 (+ 1214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.625708 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:06.625808 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 22:05:06.626038 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13024,"thread_start_us":13960,"threads_started":1}
I20260927 22:05:06.627426 18434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda78f1feaa14d79a8a2a9b39f59e2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:06.628383 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.603627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.3.159.1:45755)
0927 22:05:06.617062 (+ 13435us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:06.617076 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:06.617115 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:05:06.617229 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:06.617236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:06.617637 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:06.617905 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:06.618705 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.619864 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.623488 (+ 3624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.624638 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.624846 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.625158 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:06.627664 (+ 2506us) server_negotiation.cc:299] Negotiation successful
0927 22:05:06.627901 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13239,"thread_start_us":13046,"threads_started":1}
I20260927 22:05:06.631634 18566 tablet_service.cc:1467] Processing CreateTablet for tablet cda78f1feaa14d79a8a2a9b39f59e2f9 (DEFAULT_TABLE table=test-table [id=b3306aee226841d8ac7014f3d9ca742b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:06.632676 18302 tablet_service.cc:1467] Processing CreateTablet for tablet cda78f1feaa14d79a8a2a9b39f59e2f9 (DEFAULT_TABLE table=test-table [id=b3306aee226841d8ac7014f3d9ca742b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:06.633662 18566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda78f1feaa14d79a8a2a9b39f59e2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:06.634779 18302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cda78f1feaa14d79a8a2a9b39f59e2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:06.648129 18648 tablet_bootstrap.cc:492] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: Bootstrap starting.
I20260927 22:05:06.651013 18649 tablet_bootstrap.cc:492] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: Bootstrap starting.
I20260927 22:05:06.652455 18648 tablet_bootstrap.cc:654] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:06.653821 18648 log.cc:824] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:06.654598 18650 tablet_bootstrap.cc:492] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: Bootstrap starting.
I20260927 22:05:06.656291 18649 tablet_bootstrap.cc:654] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:06.658036 18649 log.cc:824] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:06.660261 18650 tablet_bootstrap.cc:654] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:06.661763 18650 log.cc:824] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:06.662344 18648 tablet_bootstrap.cc:492] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: No bootstrap required, opened a new log
I20260927 22:05:06.662653 18649 tablet_bootstrap.cc:492] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: No bootstrap required, opened a new log
I20260927 22:05:06.662817 18648 ts_tablet_manager.cc:1397] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.001s
I20260927 22:05:06.663085 18649 ts_tablet_manager.cc:1397] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260927 22:05:06.666127 18650 tablet_bootstrap.cc:492] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: No bootstrap required, opened a new log
I20260927 22:05:06.666478 18650 ts_tablet_manager.cc:1397] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 22:05:06.672988 18648 raft_consensus.cc:348] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } }
I20260927 22:05:06.673628 18648 raft_consensus.cc:374] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:06.673748 18648 raft_consensus.cc:729] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb62c320be2e46b4b756588d7271e80d, State: Initialized, Role: FOLLOWER
I20260927 22:05:06.674593 18648 consensus_queue.cc:260] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } }
I20260927 22:05:06.674714 18649 raft_consensus.cc:348] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } }
I20260927 22:05:06.675330 18649 raft_consensus.cc:374] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:06.675515 18649 raft_consensus.cc:729] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a39b17a706f4a46923a6f80563307c2, State: Initialized, Role: FOLLOWER
I20260927 22:05:06.676682 18499 heartbeater.cc:503] Master 127.3.159.62:37253 was elected leader, sending a full tablet report...
I20260927 22:05:06.676604 18649 consensus_queue.cc:260] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } }
I20260927 22:05:06.678691 18650 raft_consensus.cc:348] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } }
I20260927 22:05:06.679240 18650 raft_consensus.cc:374] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:06.679270 18649 ts_tablet_manager.cc:1428] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260927 22:05:06.679350 18650 raft_consensus.cc:729] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bd0c16c7d14937b2b981894ad3a43f, State: Initialized, Role: FOLLOWER
I20260927 22:05:06.680359 18650 consensus_queue.cc:260] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } }
I20260927 22:05:06.680951 18648 ts_tablet_manager.cc:1428] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260927 22:05:06.683053 18631 heartbeater.cc:503] Master 127.3.159.62:37253 was elected leader, sending a full tablet report...
I20260927 22:05:06.683717 18650 ts_tablet_manager.cc:1428] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
W20260927 22:05:06.702566 18500 tablet.cc:2369] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:05:06.763880 18632 tablet.cc:2369] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:05:06.889513 18368 tablet.cc:2369] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:05:06.948818 18655 raft_consensus.cc:482] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:05:06.949208 18655 raft_consensus.cc:504] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } }
I20260927 22:05:06.951099 18656 raft_consensus.cc:482] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:05:06.951599 18655 leader_election.cc:288] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb62c320be2e46b4b756588d7271e80d (127.3.159.2:35481), 51bd0c16c7d14937b2b981894ad3a43f (127.3.159.3:40403)
I20260927 22:05:06.951543 18656 raft_consensus.cc:504] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } }
I20260927 22:05:06.962723 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.953364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:34103 (local address 127.3.159.2:35481)
0927 22:05:06.953654 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:06.953660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:06.953685 (+ 25us) server_negotiation.cc:407] Connection header received
0927 22:05:06.953794 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:06.953800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:06.953898 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:06.954034 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:06.955501 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.956677 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.959903 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.961294 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.961463 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.961575 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:06.962257 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 22:05:06.962415 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 22:05:06.965409 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.952598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:55449 (local address 127.3.159.3:40403)
0927 22:05:06.953039 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:06.953046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:06.955036 (+ 1990us) server_negotiation.cc:407] Connection header received
0927 22:05:06.955343 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:06.955348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:06.955436 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:06.955514 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:06.956561 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.958100 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.961556 (+ 3456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.963184 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.963325 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.963503 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:06.964769 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0927 22:05:06.964989 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277}
I20260927 22:05:06.966298 18656 leader_election.cc:288] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb62c320be2e46b4b756588d7271e80d (127.3.159.2:35481), 8a39b17a706f4a46923a6f80563307c2 (127.3.159.1:45755)
I20260927 22:05:06.966569 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.952333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:40403 (local address 127.3.159.1:55449)
0927 22:05:06.954922 (+ 2589us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:06.954942 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:06.955167 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:06.955799 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:06.955807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:06.955843 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:06.956335 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.956351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.958379 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.958388 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.961310 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.961328 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.964093 (+ 2765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.964103 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.964367 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.964558 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 22:05:06.965906 (+ 1348us) client_negotiation.cc:240] Negotiation successful
0927 22:05:06.966094 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2418,"mutex_wait_us":1200,"thread_start_us":935,"threads_started":1}
I20260927 22:05:06.967545 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.952319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:35481 (local address 127.3.159.1:34103)
0927 22:05:06.953180 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:06.953204 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:06.953366 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:06.954758 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:06.954764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:06.954801 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:06.955279 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.955294 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.956893 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.956898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.959622 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.959650 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.961818 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.961825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.962026 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.962064 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 22:05:06.966882 (+ 4818us) client_negotiation.cc:240] Negotiation successful
0927 22:05:06.967096 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":1,"thread_start_us":1408,"threads_started":1}
I20260927 22:05:06.967932 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda78f1feaa14d79a8a2a9b39f59e2f9" candidate_uuid: "8a39b17a706f4a46923a6f80563307c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bd0c16c7d14937b2b981894ad3a43f" is_pre_election: true
I20260927 22:05:06.968624 18586 raft_consensus.cc:2454] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a39b17a706f4a46923a6f80563307c2 in term 0.
I20260927 22:05:06.970026 18256 leader_election.cc:302] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bd0c16c7d14937b2b981894ad3a43f, 8a39b17a706f4a46923a6f80563307c2; no voters:
I20260927 22:05:06.970575 18454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda78f1feaa14d79a8a2a9b39f59e2f9" candidate_uuid: "8a39b17a706f4a46923a6f80563307c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb62c320be2e46b4b756588d7271e80d" is_pre_election: true
I20260927 22:05:06.971189 18655 raft_consensus.cc:2781] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:05:06.971391 18655 raft_consensus.cc:482] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:05:06.971520 18655 raft_consensus.cc:3037] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:06.971482 18454 raft_consensus.cc:2454] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a39b17a706f4a46923a6f80563307c2 in term 0.
I20260927 22:05:06.973865 18634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.954412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:35481 (local address 127.3.159.3:35517)
0927 22:05:06.954807 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:06.954830 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:06.954952 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:06.963851 (+ 8899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:06.963858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:06.963899 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:06.964187 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.964201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.965893 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.965901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.968223 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.968249 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.972896 (+ 4647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.972903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.973154 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.973202 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:05:06.973327 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 22:05:06.973539 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260927 22:05:06.974084 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.955246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:35517 (local address 127.3.159.2:35481)
0927 22:05:06.963029 (+ 7783us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:06.963035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:06.963055 (+ 20us) server_negotiation.cc:407] Connection header received
0927 22:05:06.963123 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:06.963129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:06.963230 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:06.963329 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:06.964385 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.965673 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.968487 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.972663 (+ 4176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.972825 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.972966 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:06.973568 (+ 602us) server_negotiation.cc:299] Negotiation successful
0927 22:05:06.973770 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7646,"thread_start_us":15625,"threads_started":1}
I20260927 22:05:06.974229 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.963566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:40409 (local address 127.3.159.1:45755)
0927 22:05:06.963787 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:06.963793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:06.966468 (+ 2675us) server_negotiation.cc:407] Connection header received
0927 22:05:06.966697 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:06.966703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:06.966804 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:06.966939 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:06.967983 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.969130 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.971702 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.973121 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.973316 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.973455 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:06.973812 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 22:05:06.973962 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 22:05:06.975025 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:06.961855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:45755 (local address 127.3.159.3:40409)
0927 22:05:06.966350 (+ 4495us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:06.966373 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:06.966538 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:06.967115 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:06.967121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:06.967152 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:06.967762 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.967775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.969411 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.969416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.971496 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:06.971514 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:06.973403 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:06.973409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:06.973600 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:06.973633 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:05:06.974499 (+ 866us) client_negotiation.cc:240] Negotiation successful
0927 22:05:06.974670 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4322,"thread_start_us":4068,"threads_started":1}
I20260927 22:05:06.974911 18454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda78f1feaa14d79a8a2a9b39f59e2f9" candidate_uuid: "51bd0c16c7d14937b2b981894ad3a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb62c320be2e46b4b756588d7271e80d" is_pre_election: true
I20260927 22:05:06.975236 18454 raft_consensus.cc:2454] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51bd0c16c7d14937b2b981894ad3a43f in term 0.
I20260927 22:05:06.975171 18655 raft_consensus.cc:504] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } }
I20260927 22:05:06.975894 18519 leader_election.cc:302] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bd0c16c7d14937b2b981894ad3a43f, fb62c320be2e46b4b756588d7271e80d; no voters:
I20260927 22:05:06.976344 18322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda78f1feaa14d79a8a2a9b39f59e2f9" candidate_uuid: "51bd0c16c7d14937b2b981894ad3a43f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a39b17a706f4a46923a6f80563307c2" is_pre_election: true
I20260927 22:05:06.976559 18454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda78f1feaa14d79a8a2a9b39f59e2f9" candidate_uuid: "8a39b17a706f4a46923a6f80563307c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb62c320be2e46b4b756588d7271e80d"
I20260927 22:05:06.976559 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cda78f1feaa14d79a8a2a9b39f59e2f9" candidate_uuid: "8a39b17a706f4a46923a6f80563307c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51bd0c16c7d14937b2b981894ad3a43f"
I20260927 22:05:06.976836 18586 raft_consensus.cc:3037] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:06.976836 18454 raft_consensus.cc:3037] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:06.976922 18322 raft_consensus.cc:2379] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 51bd0c16c7d14937b2b981894ad3a43f in current term 1: Already voted for candidate 8a39b17a706f4a46923a6f80563307c2 in this term.
I20260927 22:05:06.977869 18655 leader_election.cc:288] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [CANDIDATE]: Term 1 election: Requested vote from peers fb62c320be2e46b4b756588d7271e80d (127.3.159.2:35481), 51bd0c16c7d14937b2b981894ad3a43f (127.3.159.3:40403)
I20260927 22:05:06.980345 18586 raft_consensus.cc:2454] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a39b17a706f4a46923a6f80563307c2 in term 1.
I20260927 22:05:06.980685 18656 raft_consensus.cc:2741] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 22:05:06.980839 18454 raft_consensus.cc:2454] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a39b17a706f4a46923a6f80563307c2 in term 1.
I20260927 22:05:06.980945 18256 leader_election.cc:302] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51bd0c16c7d14937b2b981894ad3a43f, 8a39b17a706f4a46923a6f80563307c2; no voters:
I20260927 22:05:06.981417 18655 raft_consensus.cc:2781] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:05:06.982101 18655 raft_consensus.cc:686] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 1 LEADER]: Becoming Leader. State: Replica: 8a39b17a706f4a46923a6f80563307c2, State: Running, Role: LEADER
I20260927 22:05:06.982640 18655 consensus_queue.cc:237] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } } peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } }
I20260927 22:05:06.988359 18181 catalog_manager.cc:5495] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a39b17a706f4a46923a6f80563307c2 (127.3.159.1). New cstate: current_term: 1 leader_uuid: "8a39b17a706f4a46923a6f80563307c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a39b17a706f4a46923a6f80563307c2" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 } health_report { overall_health: UNKNOWN } } }
I20260927 22:05:07.088862 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:07.079999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.3.159.1:45755)
0927 22:05:07.080331 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:07.080337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:07.080580 (+ 243us) server_negotiation.cc:407] Connection header received
0927 22:05:07.081082 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:07.081086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:07.081223 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:07.081749 (+ 526us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:05:07.082410 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:07.083323 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:07.085259 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:07.086022 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:07.086186 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:07.087107 (+ 921us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:05:07.087739 (+ 632us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:05:07.088123 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:07.088213 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 22:05:07.088514 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260927 22:05:07.100263 18586 raft_consensus.cc:1261] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 1 FOLLOWER]: Refusing update from remote peer 8a39b17a706f4a46923a6f80563307c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:05:07.100606 18454 raft_consensus.cc:1261] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d [term 1 FOLLOWER]: Refusing update from remote peer 8a39b17a706f4a46923a6f80563307c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:05:07.101438 18661 consensus_queue.cc:1035] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bd0c16c7d14937b2b981894ad3a43f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:05:07.102010 18655 consensus_queue.cc:1035] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb62c320be2e46b4b756588d7271e80d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:05:07.118407 18665 mvcc.cc:205] Tried to move back new op lower bound from 7334079312270487552 to 7334079311805145088. Current Snapshot: MvccSnapshot[applied={T|T < 7334079312270487552}]
I20260927 22:05:07.119261 18667 mvcc.cc:205] Tried to move back new op lower bound from 7334079312270487552 to 7334079311805145088. Current Snapshot: MvccSnapshot[applied={T|T < 7334079312270487552}]
I20260927 22:05:07.142652 18181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:05:07.143126 18181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:05:07.148149 18181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Sending DeleteTablet for 3 replicas of tablet cda78f1feaa14d79a8a2a9b39f59e2f9
I20260927 22:05:07.149998 18566 tablet_service.cc:1514] Processing DeleteTablet for tablet cda78f1feaa14d79a8a2a9b39f59e2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:07 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:34558
I20260927 22:05:07.150082 18434 tablet_service.cc:1514] Processing DeleteTablet for tablet cda78f1feaa14d79a8a2a9b39f59e2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:07 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:56016
I20260927 22:05:07.150524 18302 tablet_service.cc:1514] Processing DeleteTablet for tablet cda78f1feaa14d79a8a2a9b39f59e2f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:07 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:43922
I20260927 22:05:07.150806 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 18218
I20260927 22:05:07.152225 18676 tablet_replica.cc:330] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: stopping tablet replica
I20260927 22:05:07.152861 18676 raft_consensus.cc:2229] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:05:07.153643 18676 raft_consensus.cc:2258] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:07.154728 18677 tablet_replica.cc:330] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: stopping tablet replica
I20260927 22:05:07.155413 18677 raft_consensus.cc:2229] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:05:07.156509 18677 raft_consensus.cc:2258] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:07.157907 18675 tablet_replica.cc:330] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: stopping tablet replica
I20260927 22:05:07.158900 18675 raft_consensus.cc:2229] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:05:07.159559 18675 raft_consensus.cc:2258] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:07.161190 18677 ts_tablet_manager.cc:1905] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:07.162148 18675 ts_tablet_manager.cc:1905] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:07.163251 18676 ts_tablet_manager.cc:1905] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:07.171065 18677 ts_tablet_manager.cc:1918] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:07.171072 18675 ts_tablet_manager.cc:1918] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:07.171298 18675 log.cc:1196] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/cda78f1feaa14d79a8a2a9b39f59e2f9
I20260927 22:05:07.171584 18677 log.cc:1196] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/cda78f1feaa14d79a8a2a9b39f59e2f9
I20260927 22:05:07.171888 18675 ts_tablet_manager.cc:1939] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 51bd0c16c7d14937b2b981894ad3a43f: Deleting consensus metadata
I20260927 22:05:07.172514 18677 ts_tablet_manager.cc:1939] T cda78f1feaa14d79a8a2a9b39f59e2f9 P 8a39b17a706f4a46923a6f80563307c2: Deleting consensus metadata
I20260927 22:05:07.174247 18166 catalog_manager.cc:4841] TS 51bd0c16c7d14937b2b981894ad3a43f (127.3.159.3:40403): tablet cda78f1feaa14d79a8a2a9b39f59e2f9 (table test-table [id=b3306aee226841d8ac7014f3d9ca742b]) successfully deleted
I20260927 22:05:07.175107 18165 catalog_manager.cc:4841] TS 8a39b17a706f4a46923a6f80563307c2 (127.3.159.1:45755): tablet cda78f1feaa14d79a8a2a9b39f59e2f9 (table test-table [id=b3306aee226841d8ac7014f3d9ca742b]) successfully deleted
I20260927 22:05:07.176509 18676 ts_tablet_manager.cc:1918] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:07.176779 18676 log.cc:1196] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/cda78f1feaa14d79a8a2a9b39f59e2f9
I20260927 22:05:07.177595 18676 ts_tablet_manager.cc:1939] T cda78f1feaa14d79a8a2a9b39f59e2f9 P fb62c320be2e46b4b756588d7271e80d: Deleting consensus metadata
I20260927 22:05:07.180238 18165 catalog_manager.cc:4841] TS fb62c320be2e46b4b756588d7271e80d (127.3.159.2:35481): tablet cda78f1feaa14d79a8a2a9b39f59e2f9 (table test-table [id=b3306aee226841d8ac7014f3d9ca742b]) successfully deleted
I20260927 22:05:07.200239 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:07.151874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.3.159.1:45755)
0927 22:05:07.152162 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:07.152171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:07.174696 (+ 22525us) server_negotiation.cc:407] Connection header received
0927 22:05:07.175503 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:07.175509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:07.175643 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:07.175795 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:07.178704 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:07.180684 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:07.183328 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:07.184096 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:07.184265 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:07.188750 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:07.188772 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:07.188794 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:07.188844 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:07.195126 (+ 6282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:07.196866 (+ 1740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:07.196877 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:07.196880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:07.197174 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:07.197918 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:07.197922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:07.197924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:07.198556 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:07.198896 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:07.199466 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 22:05:07.199733 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 22:05:07.201581 18362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:07.323962 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 18218
I20260927 22:05:07.346114 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 18371
I20260927 22:05:07.366519 18643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:07.347448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.3.159.2:35481)
0927 22:05:07.347681 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:07.347687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:07.347718 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:05:07.347966 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:07.347972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:07.348092 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:07.348274 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:07.350146 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:07.351014 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:07.352838 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:07.353366 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:07.353622 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:07.357156 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:07.357181 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:07.357204 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:07.357267 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:07.361363 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:07.362847 (+ 1484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:07.362860 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:07.362863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:07.363101 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:07.363955 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:07.363960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:07.363963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:07.364556 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:07.364891 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:07.365693 (+ 802us) server_negotiation.cc:299] Negotiation successful
0927 22:05:07.366120 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:05:07.367537 18494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:07.490301 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 18371
I20260927 22:05:07.512125 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 18503
I20260927 22:05:07.533835 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:07.513161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.3.159.3:40403)
0927 22:05:07.513889 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:07.513894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:07.513915 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:05:07.513992 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:07.513997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:07.514073 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:07.514203 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:07.516492 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:07.517655 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:07.520127 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:07.520735 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:07.520905 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:07.525842 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:07.525862 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:07.525886 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:07.525942 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:07.529802 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:07.531103 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:07.531113 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:07.531115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:07.531402 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:07.531919 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:07.531923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:07.531925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:07.532435 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:07.532729 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:07.533126 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 22:05:07.533356 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":420,"threads_started":1}
I20260927 22:05:07.534570 18626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:07.638060 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 18503
I20260927 22:05:07.660920 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 18148
I20260927 22:05:07.662590 18211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:07.808485 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 18148
I20260927 22:05:07.840361 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37253
--webserver_interface=127.3.159.62
--webserver_port=43471
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:05:08.310652 18684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:08.311133 18684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:08.311252 18684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:08.320823 18684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:05:08.321012 18684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:08.321067 18684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:08.321132 18684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:05:08.321175 18684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:05:08.334277 18684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:37253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=43471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:08.336406 18684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:08.339094 18684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:08.360548 18689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:08.360548 18692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:08.360809 18690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:08.364301 18684 server_base.cc:1034] running on GCE node
I20260927 22:05:08.366336 18684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:08.368573 18684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:08.370398 18684 hybrid_clock.cc:648] HybridClock initialized: now 1790546708370223 us; error 117 us; skew 500 ppm
I20260927 22:05:08.371119 18684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:08 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546708, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:08.377075 18684 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:05:08.380430 18684 webserver.cc:460] Webserver started at http://127.3.159.62:43471/ using document root <none> and password file <none>
I20260927 22:05:08.381853 18684 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:08.382103 18684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:08.390285 18684 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 22:05:08.396276 18699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:08.399260 18684 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260927 22:05:08.399629 18684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "4d54427316934863a1174cf933f9fd46"
format_stamp: "Formatted at 2026-09-27 22:05:00 on dist-test-slave-sg4j"
server_key: "0ad165fcfc68f17edebc71cc9a0eb9f8"
server_key_iv: "dbdb96b54963dbe1fa82f044bed0d2f6"
server_key_version: "encryptionkey@0"
I20260927 22:05:08.401196 18684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:08.443733 18684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:08.451606 18684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:08.452039 18684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:08.485636 18684 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:37253
I20260927 22:05:08.485664 18750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:37253 every 8 connection(s)
I20260927 22:05:08.487898 18684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:05:08.490803 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 18684
I20260927 22:05:08.492254 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:45755
--local_ip_for_outbound_sockets=127.3.159.1
--tserver_master_addrs=127.3.159.62:37253
--webserver_port=45643
--webserver_interface=127.3.159.1
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:05:08.505283 18751 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:05:08.512837 18751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Bootstrap starting.
I20260927 22:05:08.545811 18751 log.cc:824] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:08.568681 18751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:05:08.569617 18751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46: Bootstrap complete.
I20260927 22:05:08.582026 18751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:08.583285 18751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d54427316934863a1174cf933f9fd46, State: Initialized, Role: FOLLOWER
I20260927 22:05:08.584352 18751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:08.584695 18751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:05:08.584928 18751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:05:08.585173 18751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:05:08.591064 18751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:08.591885 18751 leader_election.cc:302] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d54427316934863a1174cf933f9fd46; no voters:
I20260927 22:05:08.592777 18751 leader_election.cc:288] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:05:08.592960 18755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:05:08.594648 18755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [term 3 LEADER]: Becoming Leader. State: Replica: 4d54427316934863a1174cf933f9fd46, State: Running, Role: LEADER
I20260927 22:05:08.595472 18755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } }
I20260927 22:05:08.595983 18751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:05:08.599049 18757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d54427316934863a1174cf933f9fd46. Latest consensus state: current_term: 3 leader_uuid: "4d54427316934863a1174cf933f9fd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } } }
I20260927 22:05:08.599668 18757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: This master's current role is: LEADER
I20260927 22:05:08.600306 18756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4d54427316934863a1174cf933f9fd46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d54427316934863a1174cf933f9fd46" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37253 } } }
I20260927 22:05:08.600646 18756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d54427316934863a1174cf933f9fd46 [sys.catalog]: This master's current role is: LEADER
I20260927 22:05:08.613061 18762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:05:08.625926 18762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:05:08.630560 18762 catalog_manager.cc:1259] Loaded cluster ID: 9a474ea08ee94886bd32d423564d4fed
I20260927 22:05:08.630795 18762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 22:05:08.636372 18762 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcc2796f9ad google::LogMessage::Fail() at ??:0
@ 0x7fcc2797288a google::LogMessage::SendToLog() at ??:0
@ 0x7fcc2796f4de google::LogMessage::Flush() at ??:0
@ 0x7fcc27970a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcc3d3a6512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcc3d3a3976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcc3d42b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcc3d42b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcc2a5bb345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcc2a5be91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fcc2a5be799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fcc2a599eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcc2d8736db start_thread at ??:0
@ 0x7fcc2680961f clone at ??:0
W20260927 22:05:09.048080 18753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:09.048557 18753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:09.048739 18753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:09.058881 18753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:09.059154 18753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:09.059441 18753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:05:09.072580 18753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:45755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=45643
--tserver_master_addrs=127.3.159.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:09.074498 18753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:09.076356 18753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:09.096750 18792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:09.096836 18793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:09.096836 18795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:09.098392 18753 server_base.cc:1034] running on GCE node
I20260927 22:05:09.099256 18753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:09.100634 18753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:09.104264 18753 hybrid_clock.cc:648] HybridClock initialized: now 1790546709104070 us; error 44 us; skew 500 ppm
I20260927 22:05:09.104836 18753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:09 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546709, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:09.115368 18753 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:05:09.117982 18753 webserver.cc:460] Webserver started at http://127.3.159.1:45643/ using document root <none> and password file <none>
I20260927 22:05:09.118948 18753 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:09.119133 18753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:09.127185 18753 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260927 22:05:09.131587 18802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:09.134064 18753 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260927 22:05:09.134336 18753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "8a39b17a706f4a46923a6f80563307c2"
format_stamp: "Formatted at 2026-09-27 22:05:01 on dist-test-slave-sg4j"
server_key: "9deef8e4ecd289aa1b61da00f18b3556"
server_key_iv: "b5f08114eb7b14f3775ea7900d8b1081"
server_key_version: "encryptionkey@0"
I20260927 22:05:09.135565 18753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:09.175081 18753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:09.181246 18753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:09.181705 18753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:09.183211 18753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:09.185108 18753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:09.185271 18753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:09.185469 18753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:09.185585 18753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:09.230150 18914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:45755 every 8 connection(s)
I20260927 22:05:09.230150 18753 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:45755
I20260927 22:05:09.232126 18753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:05:09.234700 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 18753
I20260927 22:05:09.235528 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:35481
--local_ip_for_outbound_sockets=127.3.159.2
--tserver_master_addrs=127.3.159.62:37253
--webserver_port=36765
--webserver_interface=127.3.159.2
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:05:09.239934 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:09.236649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:52049)
0927 22:05:09.237879 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:09.238229 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":442,"threads_started":1}
W20260927 22:05:09.241145 18915 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:37253 (0 consecutive failures): Network error: Failed to ping master at 127.3.159.62:37253: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
I20260927 22:05:09.242504 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:09.241945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:56085)
0927 22:05:09.242199 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:09.242320 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 22:05:09.243958 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:09.243399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:34253)
0927 22:05:09.243625 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:09.243765 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 22:05:09.244302 18915 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:05:09.662638 18918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:09.663008 18918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:09.663175 18918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:09.672554 18918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:09.673440 18918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:09.673741 18918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:05:09.687469 18918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:35481
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=36765
--tserver_master_addrs=127.3.159.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:09.689145 18918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:09.690809 18918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:09.708813 18923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:09.708879 18926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:09.709339 18924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:09.711167 18918 server_base.cc:1034] running on GCE node
I20260927 22:05:09.712255 18918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:09.714165 18918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:09.715699 18918 hybrid_clock.cc:648] HybridClock initialized: now 1790546709715602 us; error 122 us; skew 500 ppm
I20260927 22:05:09.716205 18918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:09 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546709, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:09.720973 18918 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:05:09.723838 18918 webserver.cc:460] Webserver started at http://127.3.159.2:36765/ using document root <none> and password file <none>
I20260927 22:05:09.725032 18918 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:09.725219 18918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:09.733923 18918 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.005s
I20260927 22:05:09.738687 18933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:09.742082 18918 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260927 22:05:09.742432 18918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "fb62c320be2e46b4b756588d7271e80d"
format_stamp: "Formatted at 2026-09-27 22:05:02 on dist-test-slave-sg4j"
server_key: "662592e5f71b9d7c422052410a6b0b21"
server_key_iv: "a1fbfffc2aeecd96e2a2346041a062ef"
server_key_version: "encryptionkey@0"
I20260927 22:05:09.744446 18918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:09.765980 18918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:09.773702 18918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:09.774248 18918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:09.775805 18918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:09.777777 18918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:09.777926 18918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:09.778080 18918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:09.778174 18918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:09.827852 18918 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:35481
I20260927 22:05:09.827953 19045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:35481 every 8 connection(s)
I20260927 22:05:09.830173 18918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:05:09.834049 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 18918
I20260927 22:05:09.834969 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:40403
--local_ip_for_outbound_sockets=127.3.159.3
--tserver_master_addrs=127.3.159.62:37253
--webserver_port=45143
--webserver_interface=127.3.159.3
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:05:09.838065 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:09.834768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:39849)
0927 22:05:09.836353 (+ 1585us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:09.836723 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":572,"threads_started":1}
W20260927 22:05:09.839326 19046 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:37253 (0 consecutive failures): Network error: Failed to ping master at 127.3.159.62:37253: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
I20260927 22:05:09.841549 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:09.840505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:46647)
0927 22:05:09.840990 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:09.841318 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260927 22:05:09.842785 19048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:09.842339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:46835)
0927 22:05:09.842516 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:09.842624 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 22:05:09.843070 19046 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:05:10.246712 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:10.245337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:40519)
0927 22:05:10.246280 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:10.246415 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":457,"threads_started":1}
W20260927 22:05:10.330274 19049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:10.330798 19049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:10.331004 19049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:10.341864 19049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:10.342198 19049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:10.342438 19049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:05:10.357059 19049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:40403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=45143
--tserver_master_addrs=127.3.159.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:10.360000 19049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:10.362486 19049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:10.388037 19056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:10.388296 19058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:10.388316 19055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:10.389292 19049 server_base.cc:1034] running on GCE node
I20260927 22:05:10.390321 19049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:10.392122 19049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:10.393743 19049 hybrid_clock.cc:648] HybridClock initialized: now 1790546710393622 us; error 121 us; skew 500 ppm
I20260927 22:05:10.394279 19049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:10 dist-test-slave-sg4j krb5kdc[17596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546710, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:10.400205 19049 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:05:10.403066 19049 webserver.cc:460] Webserver started at http://127.3.159.3:45143/ using document root <none> and password file <none>
I20260927 22:05:10.404429 19049 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:10.404650 19049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:10.414670 19049 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 22:05:10.419960 19065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:10.423122 19049 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260927 22:05:10.423507 19049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "51bd0c16c7d14937b2b981894ad3a43f"
format_stamp: "Formatted at 2026-09-27 22:05:02 on dist-test-slave-sg4j"
server_key: "c35ce838b601a1e7ffba3944f7b5c99d"
server_key_iv: "b7f3417563b7e1188586280c8637b352"
server_key_version: "encryptionkey@0"
I20260927 22:05:10.425493 19049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:10.458005 19049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:10.464811 19049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:10.465260 19049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:10.466948 19049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:10.469164 19049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:10.469359 19049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:10.469570 19049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:10.469677 19049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:10.522146 19049 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:40403
I20260927 22:05:10.522198 19177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:40403 every 8 connection(s)
I20260927 22:05:10.524820 19049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:05:10.527884 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 19049
I20260927 22:05:10.533576 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:10.530158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:41245)
0927 22:05:10.531537 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:10.532140 (+ 603us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":522,"threads_started":1}
W20260927 22:05:10.534649 19178 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:37253 (0 consecutive failures): Network error: Failed to ping master at 127.3.159.62:37253: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
I20260927 22:05:10.536152 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:10.535612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:36085)
0927 22:05:10.535866 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:10.535978 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 22:05:10.537557 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:10.537054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:38809)
0927 22:05:10.537237 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:10.537339 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 22:05:10.537868 19178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:05:10.845316 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:10.844147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:43755)
0927 22:05:10.844881 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:10.845060 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":389,"threads_started":1}
I20260927 22:05:11.249018 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:11.247895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:40767)
0927 22:05:11.248604 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:11.248777 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":343,"threads_started":1}
I20260927 22:05:11.539892 19184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:11.538632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:39023)
0927 22:05:11.539391 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:11.539517 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":424,"threads_started":1}
I20260927 22:05:11.847932 19185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:11.846913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:48053)
0927 22:05:11.847604 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:11.847735 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":358,"threads_started":1}
I20260927 22:05:12.251705 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:12.250645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:43447)
0927 22:05:12.251316 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:12.251480 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":386,"threads_started":1}
I20260927 22:05:12.542479 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:12.541270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:43631)
0927 22:05:12.542097 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:12.542237 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":355,"threads_started":1}
I20260927 22:05:12.850620 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:12.849537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:53071)
0927 22:05:12.850282 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:12.850406 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":395,"threads_started":1}
I20260927 22:05:13.254246 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:13.253126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:59613)
0927 22:05:13.253914 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:13.254017 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":477,"threads_started":1}
I20260927 22:05:13.544945 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:13.543922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:42555)
0927 22:05:13.544559 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:13.544693 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":315,"threads_started":1}
I20260927 22:05:13.853050 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:13.851945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:38313)
0927 22:05:13.852638 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:13.852768 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":318,"threads_started":1}
I20260927 22:05:14.257576 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:14.255854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:43549)
0927 22:05:14.256760 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:14.257116 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":567,"threads_started":1}
I20260927 22:05:14.547739 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:14.546612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:46653)
0927 22:05:14.547398 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:14.547521 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":455,"threads_started":1}
I20260927 22:05:14.855834 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:14.854615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:38509)
0927 22:05:14.855391 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:14.855527 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":443,"threads_started":1}
I20260927 22:05:15.260686 19195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:15.259315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:33213)
0927 22:05:15.260223 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:15.260495 (+ 272us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":553,"threads_started":1}
I20260927 22:05:15.551136 19196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:15.549760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:55127)
0927 22:05:15.550654 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:15.550801 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":416,"threads_started":1}
I20260927 22:05:15.858780 19197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:15.857565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:42115)
0927 22:05:15.858437 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:15.858559 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":521,"threads_started":1}
I20260927 22:05:16.263350 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:16.262184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:54577)
0927 22:05:16.262983 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:16.263104 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":340,"threads_started":1}
I20260927 22:05:16.553704 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:16.552660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:47839)
0927 22:05:16.553314 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:16.553486 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":342,"threads_started":1}
I20260927 22:05:16.861186 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:16.860141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:58141)
0927 22:05:16.860831 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:16.860948 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":293,"threads_started":1}
I20260927 22:05:17.266072 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:17.264835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:41281)
0927 22:05:17.265771 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:17.265887 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":473,"threads_started":1}
I20260927 22:05:17.556263 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:17.555171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:39283)
0927 22:05:17.555905 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:17.556007 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":325,"threads_started":1}
I20260927 22:05:17.863760 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:17.862685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:54759)
0927 22:05:17.863450 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:17.863573 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":437,"threads_started":1}
I20260927 22:05:18.268640 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:18.267523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:34865)
0927 22:05:18.268263 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:18.268417 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":357,"threads_started":1}
I20260927 22:05:18.558409 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:18.557452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:34133)
0927 22:05:18.558136 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:18.558236 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":343,"threads_started":1}
I20260927 22:05:18.866235 19206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:18.865229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:60269)
0927 22:05:18.865927 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:18.866046 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":352,"threads_started":1}
I20260927 22:05:19.271380 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:19.270270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:53817)
0927 22:05:19.271025 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:19.271157 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":418,"threads_started":1}
I20260927 22:05:19.561542 19208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:19.560024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:45817)
0927 22:05:19.561036 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:19.561168 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":586,"threads_started":1}
I20260927 22:05:19.868808 19209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:19.867722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:51267)
0927 22:05:19.868446 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:19.868573 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":392,"threads_started":1}
I20260927 22:05:20.274202 19210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:20.273177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:37181)
0927 22:05:20.273873 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:20.273997 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":427,"threads_started":1}
I20260927 22:05:20.564432 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:20.563347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:49581)
0927 22:05:20.564056 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:20.564217 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":361,"threads_started":1}
I20260927 22:05:20.872045 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:20.870430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:56647)
0927 22:05:20.871478 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:20.871619 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":561,"threads_started":1}
I20260927 22:05:21.277999 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:21.276225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:36039)
0927 22:05:21.277273 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:21.277684 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":614,"threads_started":1}
I20260927 22:05:21.567281 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:21.566204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:55761)
0927 22:05:21.566893 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:21.567031 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":303,"threads_started":1}
I20260927 22:05:21.874908 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:21.873807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:43441)
0927 22:05:21.874485 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:21.874610 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":373,"threads_started":1}
I20260927 22:05:22.281796 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:22.280364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:44609)
0927 22:05:22.281068 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:22.281210 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":353,"threads_started":1}
I20260927 22:05:22.570014 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:22.568914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:38111)
0927 22:05:22.569612 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:22.569802 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":319,"threads_started":1}
I20260927 22:05:22.877990 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:22.876514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:44363)
0927 22:05:22.877240 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:22.877709 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":381,"threads_started":1}
I20260927 22:05:23.284564 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:23.283425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:41085)
0927 22:05:23.284131 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:23.284278 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":354,"threads_started":1}
I20260927 22:05:23.572639 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:23.571497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:42955)
0927 22:05:23.572200 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:23.572383 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":335,"threads_started":1}
I20260927 22:05:23.880661 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:23.879463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:59961)
0927 22:05:23.880291 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:23.880413 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":519,"threads_started":1}
I20260927 22:05:24.287292 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:24.286355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:42515)
0927 22:05:24.286959 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:24.287076 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":326,"threads_started":1}
I20260927 22:05:24.575090 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:24.574119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:33967)
0927 22:05:24.574756 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:24.574876 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":307,"threads_started":1}
I20260927 22:05:24.883623 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:24.882586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.2:42157)
0927 22:05:24.883247 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:24.883392 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":331,"threads_started":1}
I20260927 22:05:25.289566 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:25.288609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.1:55621)
0927 22:05:25.289232 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:25.289354 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":294,"threads_started":1}
I20260927 22:05:25.533650 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 18753
I20260927 22:05:25.569990 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:25.535429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.3.159.1:45755)
0927 22:05:25.536458 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:25.536480 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:25.536540 (+ 60us) server_negotiation.cc:407] Connection header received
0927 22:05:25.539766 (+ 3226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:25.539782 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:25.540716 (+ 934us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:25.541321 (+ 605us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:25.543543 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:25.545713 (+ 2170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:25.547641 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:25.548536 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:25.548936 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:25.560454 (+ 11518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:25.560469 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:25.560495 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:25.560595 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:25.564745 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:25.566088 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:25.566131 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:25.566140 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:25.566397 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:25.567032 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:25.567038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:25.567041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:25.567937 (+ 896us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:25.568309 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:25.568861 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 22:05:25.569588 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":500,"threads_started":1}
I20260927 22:05:25.572254 18910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:25.578183 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:25.576919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37253 (local address 127.3.159.3:45391)
0927 22:05:25.577643 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:25.577855 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:37253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":445,"threads_started":1}
I20260927 22:05:25.675815 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 18753
I20260927 22:05:25.696676 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 18918
I20260927 22:05:25.720017 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:25.698157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.3.159.2:35481)
0927 22:05:25.698779 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:25.698795 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:25.698841 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:05:25.700456 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:25.700467 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:25.701029 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:25.701568 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:25.703267 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:25.704647 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:25.706360 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:25.707230 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:25.707426 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:25.710984 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:25.710995 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:25.711016 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:25.711071 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:25.714882 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:25.716104 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:25.716112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:25.716129 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:25.716379 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:25.717025 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:25.717036 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:25.717039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:25.718184 (+ 1145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:25.718554 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:25.719206 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 22:05:25.719683 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":298,"threads_started":1}
I20260927 22:05:25.721673 19041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:25.821811 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 18918
I20260927 22:05:25.844197 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 19049
I20260927 22:05:25.868871 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:25.845508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.3.159.3:40403)
0927 22:05:25.846418 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:25.846434 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:25.846500 (+ 66us) server_negotiation.cc:407] Connection header received
0927 22:05:25.848477 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:25.848489 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:25.849018 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:25.849521 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:25.851633 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:25.853018 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:25.854914 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:25.855982 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:25.856193 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:25.860107 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:25.860121 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:25.860141 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:25.860209 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:25.864009 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:25.865162 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:25.865171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:25.865180 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:25.865536 (+ 356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:25.866134 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:25.866139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:25.866141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:25.867005 (+ 864us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:25.867318 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:25.867953 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 22:05:25.868521 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":464,"threads_started":1}
I20260927 22:05:25.870453 19173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:25.972592 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 19049
2026-09-27T22:05:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (27387 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:05:26 dist-test-slave-sg4j krb5kdc[19235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:05:26 dist-test-slave-sg4j krb5kdc[19235](info): set up 2 sockets
Sep 27 22:05:26 dist-test-slave-sg4j krb5kdc[19235](info): commencing operation
krb5kdc: starting...
W20260927 22:05:27.138489 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.022s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:05:27 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:05:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:05:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:05:27.442934 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38479
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:38479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:05:27.886989 19248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:27.887450 19248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:27.887643 19248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:27.896188 19248 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:05:27.896379 19248 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:05:27.896457 19248 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:05:27.896556 19248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:05:27.896661 19248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:27.896808 19248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:27.896894 19248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:05:27.896987 19248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:05:27.909210 19248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:38479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:27.911144 19248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:27.912925 19248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:27.928711 19253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:27.928745 19256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:27.928840 19254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:27.935619 19248 server_base.cc:1034] running on GCE node
I20260927 22:05:27.936959 19248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:27.939109 19248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:27.940461 19248 hybrid_clock.cc:648] HybridClock initialized: now 1790546727940370 us; error 59 us; skew 500 ppm
I20260927 22:05:27.940981 19248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:27 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546727, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:27.945524 19248 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:05:27.948232 19248 webserver.cc:460] Webserver started at http://127.3.159.62:41265/ using document root <none> and password file <none>
I20260927 22:05:27.949548 19248 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:27.949745 19248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:27.950207 19248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:05:27.954535 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "c075a36c4c7f4ab8b6182776a380861a"
format_stamp: "Formatted at 2026-09-27 22:05:27 on dist-test-slave-sg4j"
server_key: "9ce40c169c2bd22749375ae1deef931e"
server_key_iv: "b160d79dba4c3a28d922d06594e37ae4"
server_key_version: "encryptionkey@0"
I20260927 22:05:27.955662 19248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "c075a36c4c7f4ab8b6182776a380861a"
format_stamp: "Formatted at 2026-09-27 22:05:27 on dist-test-slave-sg4j"
server_key: "9ce40c169c2bd22749375ae1deef931e"
server_key_iv: "b160d79dba4c3a28d922d06594e37ae4"
server_key_version: "encryptionkey@0"
I20260927 22:05:27.963361 19248 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 22:05:27.968650 19263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:27.971048 19248 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 22:05:27.971386 19248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "c075a36c4c7f4ab8b6182776a380861a"
format_stamp: "Formatted at 2026-09-27 22:05:27 on dist-test-slave-sg4j"
server_key: "9ce40c169c2bd22749375ae1deef931e"
server_key_iv: "b160d79dba4c3a28d922d06594e37ae4"
server_key_version: "encryptionkey@0"
I20260927 22:05:27.971740 19248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:28.003010 19248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:28.009537 19248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:28.009940 19248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:28.036237 19248 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:38479
I20260927 22:05:28.036254 19314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:38479 every 8 connection(s)
I20260927 22:05:28.038184 19248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:05:28.043254 19315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:28.047122 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 19248
I20260927 22:05:28.047382 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:05:28.048210 3708 external_mini_cluster.cc:1428] Setting key b6ce263cb601f80d631d70cbf4c5b934
I20260927 22:05:28.060091 19315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Bootstrap starting.
I20260927 22:05:28.064788 19315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:28.066447 19315 log.cc:824] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:05:28 dist-test-slave-sg4j krb5kdc[19235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:28.070616 19315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: No bootstrap required, opened a new log
I20260927 22:05:28.080124 19315 raft_consensus.cc:348] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:05:28.080909 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:28.051363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.3.159.62:38479)
0927 22:05:28.052536 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:28.052554 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:28.052618 (+ 64us) server_negotiation.cc:407] Connection header received
0927 22:05:28.054996 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:28.055035 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:28.055723 (+ 688us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:28.056261 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:28.058494 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:28.060491 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:28.062932 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:28.063959 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:28.064193 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:28.070141 (+ 5948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:28.070153 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:28.070179 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:28.070239 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:28.074632 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:28.075776 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:28.075794 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:28.075804 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:28.076068 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:28.076641 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:28.076645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:28.076652 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:28.077666 (+ 1014us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:28.078109 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:28.078662 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 22:05:28.079301 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":557,"threads_started":1}
I20260927 22:05:28.080586 19315 raft_consensus.cc:374] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:28.081230 19315 raft_consensus.cc:729] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c075a36c4c7f4ab8b6182776a380861a, State: Initialized, Role: FOLLOWER
I20260927 22:05:28.082212 19315 consensus_queue.cc:260] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:05:28.082523 19315 raft_consensus.cc:388] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:05:28.082718 19315 raft_consensus.cc:482] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:05:28.082935 19315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:28.085858 19315 raft_consensus.cc:504] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:05:28.086500 19315 leader_election.cc:302] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c075a36c4c7f4ab8b6182776a380861a; no voters:
I20260927 22:05:28.087261 19315 leader_election.cc:288] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:05:28.087561 19320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:05:28.088653 19320 raft_consensus.cc:686] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 1 LEADER]: Becoming Leader. State: Replica: c075a36c4c7f4ab8b6182776a380861a, State: Running, Role: LEADER
I20260927 22:05:28.089568 19320 consensus_queue.cc:237] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:05:28.090499 19315 sys_catalog.cc:564] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:05:28.093938 19321 sys_catalog.cc:455] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c075a36c4c7f4ab8b6182776a380861a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } } }
I20260927 22:05:28.093904 19322 sys_catalog.cc:455] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c075a36c4c7f4ab8b6182776a380861a. Latest consensus state: current_term: 1 leader_uuid: "c075a36c4c7f4ab8b6182776a380861a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } } }
I20260927 22:05:28.094630 19322 sys_catalog.cc:458] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: This master's current role is: LEADER
I20260927 22:05:28.094627 19321 sys_catalog.cc:458] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: This master's current role is: LEADER
I20260927 22:05:28.103382 19325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:05:28.110507 19325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:05:28.125340 19325 catalog_manager.cc:1347] Generated new cluster ID: 31d227c6f1f9408d8a917cbdb9af977b
I20260927 22:05:28.125584 19325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:05:28.163035 19325 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:05:28.165127 19325 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:05:28.176784 19325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Generated new TSK 0
I20260927 22:05:28.178252 19325 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:05:28.262562 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38479
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:05:28.695338 19341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:28.695695 19341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:28.695894 19341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:28.704280 19341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:28.704440 19341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:28.704651 19341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:05:28.717114 19341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:28.718755 19341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:28.720216 19341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:28.737478 19349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:28.737452 19347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:28.737651 19346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:28.738991 19341 server_base.cc:1034] running on GCE node
I20260927 22:05:28.740006 19341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:28.741758 19341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:28.745288 19341 hybrid_clock.cc:648] HybridClock initialized: now 1790546728745178 us; error 63 us; skew 500 ppm
I20260927 22:05:28.746009 19341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:28 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546728, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:28.750988 19341 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:05:28.753309 19341 webserver.cc:460] Webserver started at http://127.3.159.1:35173/ using document root <none> and password file <none>
I20260927 22:05:28.754374 19341 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:28.754521 19341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:28.754876 19341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:05:28.758842 19341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "50331455b4084a198b2255e05bca12a8"
format_stamp: "Formatted at 2026-09-27 22:05:28 on dist-test-slave-sg4j"
server_key: "416e2165374e96a617ae65744c527bf1"
server_key_iv: "78931a352333ac43f77f8d4e7db0f28a"
server_key_version: "encryptionkey@0"
I20260927 22:05:28.759842 19341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "50331455b4084a198b2255e05bca12a8"
format_stamp: "Formatted at 2026-09-27 22:05:28 on dist-test-slave-sg4j"
server_key: "416e2165374e96a617ae65744c527bf1"
server_key_iv: "78931a352333ac43f77f8d4e7db0f28a"
server_key_version: "encryptionkey@0"
I20260927 22:05:28.767726 19341 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 22:05:28.773478 19356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:28.776080 19341 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 22:05:28.776384 19341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "50331455b4084a198b2255e05bca12a8"
format_stamp: "Formatted at 2026-09-27 22:05:28 on dist-test-slave-sg4j"
server_key: "416e2165374e96a617ae65744c527bf1"
server_key_iv: "78931a352333ac43f77f8d4e7db0f28a"
server_key_version: "encryptionkey@0"
I20260927 22:05:28.776714 19341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:28.801172 19341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:28.807137 19341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:28.807524 19341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:28.808962 19341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:28.810782 19341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:28.810873 19341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:28.811018 19341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:28.811090 19341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:28.856473 19341 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:46743
I20260927 22:05:28.856532 19468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:46743 every 8 connection(s)
I20260927 22:05:28.858304 19341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:05:28.868198 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 19341
I20260927 22:05:28.868479 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:05:28.869195 3708 external_mini_cluster.cc:1428] Setting key 6b440b4f1d64bc8c3d844f5e667851db
Sep 27 22:05:28 dist-test-slave-sg4j krb5kdc[19235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546728, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:28.887519 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:28.861706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:56613 (local address 127.3.159.62:38479)
0927 22:05:28.862315 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:28.862322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:28.863480 (+ 1158us) server_negotiation.cc:407] Connection header received
0927 22:05:28.866196 (+ 2716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:28.866203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:28.866335 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:28.866495 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:28.868757 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:28.869848 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:28.872482 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:28.873019 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:28.873207 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:28.878980 (+ 5773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:28.878992 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:28.879000 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:28.879061 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:28.882828 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:28.884227 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:28.884232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:28.884235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:28.884399 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:28.885105 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:28.885110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:28.885114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:28.885513 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:28.885661 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:28.886849 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0927 22:05:28.887079 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":298,"threads_started":1}
I20260927 22:05:28.889271 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:28.862260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:56613)
0927 22:05:28.863235 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:28.863298 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:28.865785 (+ 2487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:28.866860 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:28.866872 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:28.867610 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:28.868486 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:28.868512 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:28.870115 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:28.870120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:28.872230 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:28.872262 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:28.873347 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:28.873358 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:28.873792 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:28.874785 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:28.874830 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:28.878571 (+ 3741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:28.883075 (+ 4504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:28.883092 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:28.883120 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:28.884021 (+ 901us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:28.884548 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:28.884557 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:28.884563 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:28.884881 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:28.885663 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:28.885676 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:28.886318 (+ 642us) client_negotiation.cc:769] Sending connection context
0927 22:05:28.886697 (+ 379us) client_negotiation.cc:240] Negotiation successful
0927 22:05:28.887457 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":379,"threads_started":1}
I20260927 22:05:28.891607 19469 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38479
I20260927 22:05:28.892217 19469 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:28.893155 19469 heartbeater.cc:511] Master 127.3.159.62:38479 requested a full tablet report, sending...
I20260927 22:05:28.895859 19280 ts_manager.cc:194] Registered new tserver with Master: 50331455b4084a198b2255e05bca12a8 (127.3.159.1:46743)
I20260927 22:05:28.897666 19280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:56613
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:05:28.946903 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38479
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:05:29.428449 19475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:29.428938 19475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:29.429136 19475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:29.440111 19475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:29.440394 19475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:29.440604 19475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:05:29.453513 19475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:29.455601 19475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:29.457614 19475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:29.480064 19480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:29.480099 19481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:29.480355 19483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:29.482400 19475 server_base.cc:1034] running on GCE node
I20260927 22:05:29.483418 19475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:29.484781 19475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:29.486320 19475 hybrid_clock.cc:648] HybridClock initialized: now 1790546729486231 us; error 68 us; skew 500 ppm
I20260927 22:05:29.486960 19475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:29 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546729, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:29.492127 19475 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:05:29.495492 19475 webserver.cc:460] Webserver started at http://127.3.159.2:38603/ using document root <none> and password file <none>
I20260927 22:05:29.497695 19475 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:29.497939 19475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:29.498359 19475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:05:29.504352 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "6399b7a7faab4109a65c59e36d7e19ed"
format_stamp: "Formatted at 2026-09-27 22:05:29 on dist-test-slave-sg4j"
server_key: "2bfb6747c4227d71e8d47648206b8f43"
server_key_iv: "5d1a74d4f07630f8c1182ec841b38c3b"
server_key_version: "encryptionkey@0"
I20260927 22:05:29.505592 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "6399b7a7faab4109a65c59e36d7e19ed"
format_stamp: "Formatted at 2026-09-27 22:05:29 on dist-test-slave-sg4j"
server_key: "2bfb6747c4227d71e8d47648206b8f43"
server_key_iv: "5d1a74d4f07630f8c1182ec841b38c3b"
server_key_version: "encryptionkey@0"
I20260927 22:05:29.513087 19475 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260927 22:05:29.519526 19490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:29.521914 19475 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:05:29.522218 19475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "6399b7a7faab4109a65c59e36d7e19ed"
format_stamp: "Formatted at 2026-09-27 22:05:29 on dist-test-slave-sg4j"
server_key: "2bfb6747c4227d71e8d47648206b8f43"
server_key_iv: "5d1a74d4f07630f8c1182ec841b38c3b"
server_key_version: "encryptionkey@0"
I20260927 22:05:29.522502 19475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:29.555034 19475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:29.560901 19475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:29.561354 19475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:29.562896 19475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:29.565223 19475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:29.565392 19475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:29.565611 19475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:29.565701 19475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:29.611143 19475 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:40879
I20260927 22:05:29.611287 19602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:40879 every 8 connection(s)
I20260927 22:05:29.612700 19475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:05:29.613147 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 19475
I20260927 22:05:29.613451 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:05:29.614711 3708 external_mini_cluster.cc:1428] Setting key 01d14d6dee08575bc2fe5c620a41a569
Sep 27 22:05:29 dist-test-slave-sg4j krb5kdc[19235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546729, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:29.640561 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:29.617134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:33281 (local address 127.3.159.62:38479)
0927 22:05:29.617855 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:29.617862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:29.619153 (+ 1291us) server_negotiation.cc:407] Connection header received
0927 22:05:29.621754 (+ 2601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:29.621760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:29.621878 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:29.622022 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:29.623769 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:29.624574 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:29.626305 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:29.626857 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:29.627002 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:29.632275 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:29.632282 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:29.632286 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:29.632344 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:29.636117 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:29.637352 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:29.637358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:29.637361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:29.637614 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:29.638185 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:29.638191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:29.638205 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:29.638567 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:29.638767 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:29.639752 (+ 985us) server_negotiation.cc:299] Negotiation successful
0927 22:05:29.640097 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":399,"threads_started":1}
I20260927 22:05:29.642242 19606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:29.617909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:33281)
0927 22:05:29.618919 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:29.618971 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:29.621186 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:29.622309 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:29.622318 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:29.622927 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:29.623560 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:29.623579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:29.624760 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:29.624764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:29.626141 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:29.626155 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:29.627121 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:29.627127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:29.627392 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:29.628394 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:29.628419 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:29.631875 (+ 3456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:29.636429 (+ 4554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:29.636443 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:29.636466 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:29.637146 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:29.637792 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:29.637796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:29.637799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:29.638040 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:29.638731 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:29.638742 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:29.639277 (+ 535us) client_negotiation.cc:769] Sending connection context
0927 22:05:29.639628 (+ 351us) client_negotiation.cc:240] Negotiation successful
0927 22:05:29.640370 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":416,"threads_started":1}
I20260927 22:05:29.644708 19603 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38479
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
I20260927 22:05:29.645313 19603 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:29.646842 19603 heartbeater.cc:511] Master 127.3.159.62:38479 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
I20260927 22:05:29.649658 19280 ts_manager.cc:194] Registered new tserver with Master: 6399b7a7faab4109a65c59e36d7e19ed (127.3.159.2:40879)
I20260927 22:05:29.650794 19280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:33281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:05:29.686504 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38479
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:05:29.902962 19469 heartbeater.cc:503] Master 127.3.159.62:38479 was elected leader, sending a full tablet report...
W20260927 22:05:30.140921 19609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:30.141327 19609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:30.141535 19609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:30.150537 19609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:30.150699 19609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:30.150820 19609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:05:30.162786 19609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:30.164408 19609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:30.165922 19609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:30.183830 19617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:30.183765 19615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:30.183859 19614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:30.185489 19609 server_base.cc:1034] running on GCE node
I20260927 22:05:30.186517 19609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:30.188069 19609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:30.191852 19609 hybrid_clock.cc:648] HybridClock initialized: now 1790546730191522 us; error 283 us; skew 500 ppm
I20260927 22:05:30.192611 19609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:30 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546730, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:30.197259 19609 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:05:30.199601 19609 webserver.cc:460] Webserver started at http://127.3.159.3:46455/ using document root <none> and password file <none>
I20260927 22:05:30.200678 19609 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:30.200814 19609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:30.201223 19609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:05:30.205190 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "32926f9156864f04a58b93c4b6b9078a"
format_stamp: "Formatted at 2026-09-27 22:05:30 on dist-test-slave-sg4j"
server_key: "e6402182a75ee53b28fd349d3ab3ae1d"
server_key_iv: "8705159ac96266c07818456c3d6dd096"
server_key_version: "encryptionkey@0"
I20260927 22:05:30.206305 19609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "32926f9156864f04a58b93c4b6b9078a"
format_stamp: "Formatted at 2026-09-27 22:05:30 on dist-test-slave-sg4j"
server_key: "e6402182a75ee53b28fd349d3ab3ae1d"
server_key_iv: "8705159ac96266c07818456c3d6dd096"
server_key_version: "encryptionkey@0"
I20260927 22:05:30.213824 19609 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:05:30.219069 19624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:30.221290 19609 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260927 22:05:30.221628 19609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "32926f9156864f04a58b93c4b6b9078a"
format_stamp: "Formatted at 2026-09-27 22:05:30 on dist-test-slave-sg4j"
server_key: "e6402182a75ee53b28fd349d3ab3ae1d"
server_key_iv: "8705159ac96266c07818456c3d6dd096"
server_key_version: "encryptionkey@0"
I20260927 22:05:30.221907 19609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:30.264127 19609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:30.271603 19609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:30.272012 19609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:30.273658 19609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:30.275714 19609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:30.275821 19609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:30.275954 19609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:30.275993 19609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:30.323904 19609 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:36771
I20260927 22:05:30.323984 19736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:36771 every 8 connection(s)
I20260927 22:05:30.325759 19609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:05:30.334890 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 19609
I20260927 22:05:30.335186 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:05:30.335731 3708 external_mini_cluster.cc:1428] Setting key cc6a0ba88d74cf1102d71eb710998437
Sep 27 22:05:30 dist-test-slave-sg4j krb5kdc[19235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546730, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:30.354218 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.329652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:36975 (local address 127.3.159.62:38479)
0927 22:05:30.330221 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:30.330226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:30.331667 (+ 1441us) server_negotiation.cc:407] Connection header received
0927 22:05:30.334318 (+ 2651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:30.334324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:30.334492 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:30.334629 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:30.336540 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.337622 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.340043 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.340776 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.340985 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.346198 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:30.346204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:30.346207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:30.346253 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:30.349292 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:30.350263 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:30.350272 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:30.350276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:30.350423 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:30.351066 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:30.351076 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:30.351080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:30.351749 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:30.351985 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:30.353072 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0927 22:05:30.353696 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":313,"threads_started":1}
I20260927 22:05:30.355934 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.330318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:36975)
0927 22:05:30.331406 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:30.331459 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:30.333944 (+ 2485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:30.334920 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:30.334930 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:30.335643 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:30.336339 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.336358 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.337818 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.337822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.339660 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.339684 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.341177 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.341188 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.341603 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.342575 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:30.342643 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:30.345829 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:30.349546 (+ 3717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:30.349558 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:30.349588 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:30.350115 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:30.350546 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:30.350552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:30.350555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:30.350784 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:30.351961 (+ 1177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:30.351973 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:30.352532 (+ 559us) client_negotiation.cc:769] Sending connection context
0927 22:05:30.352927 (+ 395us) client_negotiation.cc:240] Negotiation successful
0927 22:05:30.353976 (+ 1049us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":488,"threads_started":1}
I20260927 22:05:30.358306 19737 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38479
I20260927 22:05:30.358898 19737 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:30.360014 19737 heartbeater.cc:511] Master 127.3.159.62:38479 requested a full tablet report, sending...
I20260927 22:05:30.362670 19280 ts_manager.cc:194] Registered new tserver with Master: 32926f9156864f04a58b93c4b6b9078a (127.3.159.3:36771)
I20260927 22:05:30.364490 19280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:36975
I20260927 22:05:30.368337 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:05:30.392922 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.374784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.3.159.62:38479)
0927 22:05:30.375075 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:30.375081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:30.375498 (+ 417us) server_negotiation.cc:407] Connection header received
0927 22:05:30.375808 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:30.375814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:30.375951 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:30.376100 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:30.378420 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.379444 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.381265 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.381916 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.382095 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.385556 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:30.385563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:30.385568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:30.385625 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:30.388911 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:30.390068 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:30.390074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:30.390077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:30.390270 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:30.390742 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:30.390746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:30.390748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:30.391239 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:30.391567 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:30.392072 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 22:05:30.392371 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 22:05:30.399276 19279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:05:30.408172 19279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:05:30.443199 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.427083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46743 (local address 127.0.0.1:45260)
0927 22:05:30.428706 (+ 1623us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:30.428778 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:30.429103 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:30.432959 (+ 3856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:30.432965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:30.433008 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:30.434053 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.434066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.436089 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.436097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.439959 (+ 3862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.439972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.441750 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.441757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.442030 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.442138 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 22:05:30.442335 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 22:05:30.442600 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"mutex_wait_us":627,"thread_start_us":925,"threads_started":1}
I20260927 22:05:30.443827 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.430278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:40879 (local address 127.0.0.1:44380)
0927 22:05:30.430910 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:30.430939 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:30.432129 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:30.433487 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:30.433498 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:30.433528 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:30.434492 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.434503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.436361 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.436366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.439962 (+ 3596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.439985 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.442317 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.442322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.442539 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.442583 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 22:05:30.443018 (+ 435us) client_negotiation.cc:240] Negotiation successful
0927 22:05:30.443183 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":639,"thread_start_us":398,"threads_started":1}
I20260927 22:05:30.444497 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.427573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.3.159.1:46743)
0927 22:05:30.432110 (+ 4537us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:30.432117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:30.432141 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:05:30.432287 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:30.432296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:30.432643 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:30.433773 (+ 1130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:30.434312 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.435918 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.440202 (+ 4284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.441500 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.441677 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.441991 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:30.443604 (+ 1613us) server_negotiation.cc:299] Negotiation successful
0927 22:05:30.444078 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4372,"thread_start_us":4233,"threads_started":1}
I20260927 22:05:30.444649 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.427400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:36771 (local address 127.0.0.1:38764)
0927 22:05:30.430127 (+ 2727us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:30.430157 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:30.430330 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:30.432663 (+ 2333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:30.432671 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:30.432705 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:30.433326 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.433342 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.435358 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.435366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.439450 (+ 4084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.439483 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.442276 (+ 2793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.442286 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.443834 (+ 1548us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.443886 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 22:05:30.444125 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 22:05:30.444306 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2443,"mutex_wait_us":1227,"thread_start_us":2040,"threads_started":1}
I20260927 22:05:30.445536 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.429142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38764 (local address 127.3.159.3:36771)
0927 22:05:30.431835 (+ 2693us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:30.431846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:30.431891 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:05:30.432037 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:30.432048 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:30.432404 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:30.432661 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:30.433648 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.435197 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.439760 (+ 4563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.441567 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.441738 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.442966 (+ 1228us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:30.444979 (+ 2013us) server_negotiation.cc:299] Negotiation successful
0927 22:05:30.445161 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2409,"thread_start_us":2185,"threads_started":1}
I20260927 22:05:30.446131 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.430689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.3.159.2:40879)
0927 22:05:30.431597 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:30.431607 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:30.431648 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:05:30.432379 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:30.432391 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:30.433092 (+ 701us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:30.433304 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:30.434691 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.436156 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.440249 (+ 4093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.441809 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.441936 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.442175 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:30.445561 (+ 3386us) server_negotiation.cc:299] Negotiation successful
0927 22:05:30.445776 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":500,"threads_started":1}
I20260927 22:05:30.449963 19538 tablet_service.cc:1467] Processing CreateTablet for tablet 1d534e34ad804dbb8e0bb714971a0419 (DEFAULT_TABLE table=test-table [id=a0839a08b8ca4ab3bd954aefa8a08d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:30.450344 19404 tablet_service.cc:1467] Processing CreateTablet for tablet 1d534e34ad804dbb8e0bb714971a0419 (DEFAULT_TABLE table=test-table [id=a0839a08b8ca4ab3bd954aefa8a08d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:30.450341 19672 tablet_service.cc:1467] Processing CreateTablet for tablet 1d534e34ad804dbb8e0bb714971a0419 (DEFAULT_TABLE table=test-table [id=a0839a08b8ca4ab3bd954aefa8a08d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:30.451640 19538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d534e34ad804dbb8e0bb714971a0419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:30.452354 19404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d534e34ad804dbb8e0bb714971a0419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:30.452332 19672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d534e34ad804dbb8e0bb714971a0419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:30.467819 19756 tablet_bootstrap.cc:492] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: Bootstrap starting.
I20260927 22:05:30.471791 19757 tablet_bootstrap.cc:492] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: Bootstrap starting.
I20260927 22:05:30.472309 19756 tablet_bootstrap.cc:654] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:30.470933 19758 tablet_bootstrap.cc:492] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: Bootstrap starting.
I20260927 22:05:30.473867 19756 log.cc:824] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:30.476224 19757 tablet_bootstrap.cc:654] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:30.476977 19758 tablet_bootstrap.cc:654] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:30.477913 19757 log.cc:824] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:30.478480 19758 log.cc:824] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:30.478883 19756 tablet_bootstrap.cc:492] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: No bootstrap required, opened a new log
I20260927 22:05:30.479297 19756 ts_tablet_manager.cc:1397] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260927 22:05:30.481875 19757 tablet_bootstrap.cc:492] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: No bootstrap required, opened a new log
I20260927 22:05:30.482312 19757 ts_tablet_manager.cc:1397] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260927 22:05:30.482602 19758 tablet_bootstrap.cc:492] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: No bootstrap required, opened a new log
I20260927 22:05:30.482956 19758 ts_tablet_manager.cc:1397] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260927 22:05:30.492080 19756 raft_consensus.cc:348] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } } peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } }
I20260927 22:05:30.492645 19756 raft_consensus.cc:374] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:30.492817 19756 raft_consensus.cc:729] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6399b7a7faab4109a65c59e36d7e19ed, State: Initialized, Role: FOLLOWER
I20260927 22:05:30.492548 19757 raft_consensus.cc:348] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } } peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } }
I20260927 22:05:30.493157 19757 raft_consensus.cc:374] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:30.493341 19757 raft_consensus.cc:729] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50331455b4084a198b2255e05bca12a8, State: Initialized, Role: FOLLOWER
I20260927 22:05:30.493831 19756 consensus_queue.cc:260] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } } peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } }
I20260927 22:05:30.494098 19757 consensus_queue.cc:260] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } } peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } }
I20260927 22:05:30.494792 19758 raft_consensus.cc:348] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } } peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } }
I20260927 22:05:30.495395 19758 raft_consensus.cc:374] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:30.495420 19603 heartbeater.cc:503] Master 127.3.159.62:38479 was elected leader, sending a full tablet report...
I20260927 22:05:30.495545 19758 raft_consensus.cc:729] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32926f9156864f04a58b93c4b6b9078a, State: Initialized, Role: FOLLOWER
I20260927 22:05:30.496357 19758 consensus_queue.cc:260] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } } peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } }
I20260927 22:05:30.497954 19756 ts_tablet_manager.cc:1428] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20260927 22:05:30.500545 19757 ts_tablet_manager.cc:1428] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260927 22:05:30.503131 19737 heartbeater.cc:503] Master 127.3.159.62:38479 was elected leader, sending a full tablet report...
I20260927 22:05:30.504362 19758 ts_tablet_manager.cc:1428] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: Time spent starting tablet: real 0.021s user 0.017s sys 0.003s
W20260927 22:05:30.578102 19738 tablet.cc:2369] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:05:30.613544 19470 tablet.cc:2369] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:05:30.616268 19762 raft_consensus.cc:482] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 22:05:30.616887 19604 tablet.cc:2369] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:05:30.616803 19762 raft_consensus.cc:504] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } } peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } }
I20260927 22:05:30.619326 19762 leader_election.cc:288] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50331455b4084a198b2255e05bca12a8 (127.3.159.1:46743), 32926f9156864f04a58b93c4b6b9078a (127.3.159.3:36771)
I20260927 22:05:30.628839 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.619934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:50291 (local address 127.3.159.3:36771)
0927 22:05:30.620253 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:30.620261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:30.620823 (+ 562us) server_negotiation.cc:407] Connection header received
0927 22:05:30.621096 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:30.621102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:30.621230 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:30.621344 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:30.622312 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.623620 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.626454 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.627586 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.627774 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.627905 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:30.628411 (+ 506us) server_negotiation.cc:299] Negotiation successful
0927 22:05:30.628581 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260927 22:05:30.629017 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.619767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:36771 (local address 127.3.159.2:50291)
0927 22:05:30.620679 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:30.620706 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:30.620926 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:30.621611 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:30.621617 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:30.621644 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:30.622136 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.622147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.623768 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.623774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.626197 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.626217 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.627874 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.627880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.628079 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.628122 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:05:30.628358 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 22:05:30.628564 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":471,"threads_started":1}
I20260927 22:05:30.629977 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.619965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:43011 (local address 127.3.159.1:46743)
0927 22:05:30.620411 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:30.620417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:30.621916 (+ 1499us) server_negotiation.cc:407] Connection header received
0927 22:05:30.622084 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:30.622089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:30.622178 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:30.622318 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:30.623374 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.624677 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.627277 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.628606 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.628773 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.628893 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:30.629533 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 22:05:30.629698 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316}
I20260927 22:05:30.629982 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.620745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46743 (local address 127.3.159.2:43011)
0927 22:05:30.621745 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:30.621772 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:30.621917 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:30.622368 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:30.622375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:30.622405 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:30.623035 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.623052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.624878 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.624884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.627004 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:30.627016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.629078 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.629085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:30.629307 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.629340 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 22:05:30.629479 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 22:05:30.629634 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":476,"threads_started":1}
I20260927 22:05:30.630402 19692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d534e34ad804dbb8e0bb714971a0419" candidate_uuid: "6399b7a7faab4109a65c59e36d7e19ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32926f9156864f04a58b93c4b6b9078a" is_pre_election: true
I20260927 22:05:30.631085 19692 raft_consensus.cc:2454] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6399b7a7faab4109a65c59e36d7e19ed in term 0.
I20260927 22:05:30.631659 19424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d534e34ad804dbb8e0bb714971a0419" candidate_uuid: "6399b7a7faab4109a65c59e36d7e19ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50331455b4084a198b2255e05bca12a8" is_pre_election: true
I20260927 22:05:30.631922 19492 leader_election.cc:302] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32926f9156864f04a58b93c4b6b9078a, 6399b7a7faab4109a65c59e36d7e19ed; no voters:
I20260927 22:05:30.632324 19424 raft_consensus.cc:2454] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6399b7a7faab4109a65c59e36d7e19ed in term 0.
I20260927 22:05:30.632563 19762 raft_consensus.cc:2781] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:05:30.632767 19762 raft_consensus.cc:482] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:05:30.632908 19762 raft_consensus.cc:3037] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:30.635725 19762 raft_consensus.cc:504] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } } peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } }
I20260927 22:05:30.636672 19762 leader_election.cc:288] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [CANDIDATE]: Term 1 election: Requested vote from peers 50331455b4084a198b2255e05bca12a8 (127.3.159.1:46743), 32926f9156864f04a58b93c4b6b9078a (127.3.159.3:36771)
I20260927 22:05:30.637521 19424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d534e34ad804dbb8e0bb714971a0419" candidate_uuid: "6399b7a7faab4109a65c59e36d7e19ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50331455b4084a198b2255e05bca12a8"
I20260927 22:05:30.637727 19692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d534e34ad804dbb8e0bb714971a0419" candidate_uuid: "6399b7a7faab4109a65c59e36d7e19ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32926f9156864f04a58b93c4b6b9078a"
I20260927 22:05:30.637835 19424 raft_consensus.cc:3037] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:30.638103 19692 raft_consensus.cc:3037] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:30.640615 19692 raft_consensus.cc:2454] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6399b7a7faab4109a65c59e36d7e19ed in term 1.
I20260927 22:05:30.640702 19424 raft_consensus.cc:2454] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6399b7a7faab4109a65c59e36d7e19ed in term 1.
I20260927 22:05:30.641451 19492 leader_election.cc:302] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32926f9156864f04a58b93c4b6b9078a, 6399b7a7faab4109a65c59e36d7e19ed; no voters:
I20260927 22:05:30.641942 19762 raft_consensus.cc:2781] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:05:30.642561 19762 raft_consensus.cc:686] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 1 LEADER]: Becoming Leader. State: Replica: 6399b7a7faab4109a65c59e36d7e19ed, State: Running, Role: LEADER
I20260927 22:05:30.643210 19762 consensus_queue.cc:237] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } } peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } }
I20260927 22:05:30.648722 19279 catalog_manager.cc:5495] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 6399b7a7faab4109a65c59e36d7e19ed (127.3.159.2). New cstate: current_term: 1 leader_uuid: "6399b7a7faab4109a65c59e36d7e19ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } health_report { overall_health: HEALTHY } } }
I20260927 22:05:30.698868 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.690963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.3.159.2:40879)
0927 22:05:30.691302 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:30.691308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:30.691355 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:05:30.691491 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:30.691495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:30.691607 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:30.691814 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:05:30.692425 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.693338 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.695553 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.696076 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.696229 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.697184 (+ 955us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:05:30.698044 (+ 860us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:05:30.698196 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:30.698477 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 22:05:30.698580 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260927 22:05:30.709746 19692 raft_consensus.cc:1261] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a [term 1 FOLLOWER]: Refusing update from remote peer 6399b7a7faab4109a65c59e36d7e19ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:05:30.710085 19424 raft_consensus.cc:1261] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8 [term 1 FOLLOWER]: Refusing update from remote peer 6399b7a7faab4109a65c59e36d7e19ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:05:30.710994 19767 consensus_queue.cc:1035] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:05:30.711522 19762 consensus_queue.cc:1035] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:05:30.726330 19770 mvcc.cc:205] Tried to move back new op lower bound from 7334079408974938112 to 7334079408719425536. Current Snapshot: MvccSnapshot[applied={T|T < 7334079408974938112}]
I20260927 22:05:30.747977 19280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:05:30.748345 19280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:05:30.753508 19280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Sending DeleteTablet for 3 replicas of tablet 1d534e34ad804dbb8e0bb714971a0419
I20260927 22:05:30.755155 19672 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d534e34ad804dbb8e0bb714971a0419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:30 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:38764
I20260927 22:05:30.755337 19538 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d534e34ad804dbb8e0bb714971a0419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:30 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:44380
I20260927 22:05:30.755331 19404 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d534e34ad804dbb8e0bb714971a0419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:30 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:45260
I20260927 22:05:30.756616 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 19341
I20260927 22:05:30.768024 19780 tablet_replica.cc:330] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: stopping tablet replica
I20260927 22:05:30.770071 19780 raft_consensus.cc:2229] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:05:30.771719 19782 tablet_replica.cc:330] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: stopping tablet replica
I20260927 22:05:30.772872 19780 raft_consensus.cc:2258] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:30.773880 19782 raft_consensus.cc:2229] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:05:30.776010 19782 raft_consensus.cc:2258] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:30.778450 19781 tablet_replica.cc:330] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: stopping tablet replica
I20260927 22:05:30.778925 19780 ts_tablet_manager.cc:1905] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:30.779266 19781 raft_consensus.cc:2229] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:05:30.780048 19781 raft_consensus.cc:2258] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:30.780606 19782 ts_tablet_manager.cc:1905] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:30.783201 19781 ts_tablet_manager.cc:1905] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:30.787513 19780 ts_tablet_manager.cc:1918] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:30.787897 19780 log.cc:1196] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/1d534e34ad804dbb8e0bb714971a0419
I20260927 22:05:30.788600 19780 ts_tablet_manager.cc:1939] T 1d534e34ad804dbb8e0bb714971a0419 P 32926f9156864f04a58b93c4b6b9078a: Deleting consensus metadata
I20260927 22:05:30.790171 19782 ts_tablet_manager.cc:1918] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:30.790515 19782 log.cc:1196] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/1d534e34ad804dbb8e0bb714971a0419
I20260927 22:05:30.791178 19782 ts_tablet_manager.cc:1939] T 1d534e34ad804dbb8e0bb714971a0419 P 6399b7a7faab4109a65c59e36d7e19ed: Deleting consensus metadata
I20260927 22:05:30.791901 19781 ts_tablet_manager.cc:1918] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:30.792143 19781 log.cc:1196] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/1d534e34ad804dbb8e0bb714971a0419
I20260927 22:05:30.792793 19781 ts_tablet_manager.cc:1939] T 1d534e34ad804dbb8e0bb714971a0419 P 50331455b4084a198b2255e05bca12a8: Deleting consensus metadata
Sep 27 22:05:30 dist-test-slave-sg4j krb5kdc[19235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:05:30.793926 19266 catalog_manager.cc:4841] TS 6399b7a7faab4109a65c59e36d7e19ed (127.3.159.2:40879): tablet 1d534e34ad804dbb8e0bb714971a0419 (table test-table [id=a0839a08b8ca4ab3bd954aefa8a08d25]) successfully deleted
I20260927 22:05:30.795779 19266 catalog_manager.cc:4841] TS 50331455b4084a198b2255e05bca12a8 (127.3.159.1:46743): tablet 1d534e34ad804dbb8e0bb714971a0419 (table test-table [id=a0839a08b8ca4ab3bd954aefa8a08d25]) successfully deleted
I20260927 22:05:30.795928 19265 catalog_manager.cc:4841] TS 32926f9156864f04a58b93c4b6b9078a (127.3.159.3:36771): tablet 1d534e34ad804dbb8e0bb714971a0419 (table test-table [id=a0839a08b8ca4ab3bd954aefa8a08d25]) successfully deleted
I20260927 22:05:30.806180 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.757999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.3.159.1:46743)
0927 22:05:30.758226 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:30.758233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:30.765306 (+ 7073us) server_negotiation.cc:407] Connection header received
0927 22:05:30.765754 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:30.765761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:30.765882 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:30.766115 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:30.770210 (+ 4095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.771521 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.773964 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.774726 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.774867 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.796079 (+ 21212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:30.796104 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:30.796128 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:30.796196 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:30.801467 (+ 5271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:30.802859 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:30.802871 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:30.802875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:30.803084 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:30.803792 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:30.803798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:30.803803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:30.804387 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:30.804790 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:30.805263 (+ 473us) server_negotiation.cc:299] Negotiation successful
0927 22:05:30.805708 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 22:05:30.807164 19464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:30.921633 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 19341
I20260927 22:05:30.950199 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 19475
Sep 27 22:05:30 dist-test-slave-sg4j krb5kdc[19235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:05:30.974951 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:30.951536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.3.159.2:40879)
0927 22:05:30.951849 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:30.951854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:30.952002 (+ 148us) server_negotiation.cc:407] Connection header received
0927 22:05:30.952290 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:30.952294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:30.952399 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:30.952581 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:30.954896 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.955897 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.958524 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:30.959117 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:30.959326 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:30.966624 (+ 7298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:30.966650 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:30.966680 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:30.966763 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:30.971207 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:30.972327 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:30.972337 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:30.972340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:30.972516 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:30.972946 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:30.972950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:30.972953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:30.973489 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:30.973916 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:30.974225 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 22:05:30.974518 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:05:30.975878 19598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:31.113236 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 19475
I20260927 22:05:31.147905 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 19609
Sep 27 22:05:31 dist-test-slave-sg4j krb5kdc[19235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:05:31.173085 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:31.148944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.3.159.3:36771)
0927 22:05:31.149741 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:31.149746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:31.149767 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:05:31.149834 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:31.149838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:31.149909 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:31.150036 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:31.152997 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:31.154094 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:31.157274 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:31.157849 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:31.158040 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:31.166063 (+ 8023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:31.166089 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:31.166112 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:31.166176 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:31.169735 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:31.170661 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:31.170669 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:31.170672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:31.170869 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:31.171303 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:31.171309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:31.171313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:31.171726 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:31.172046 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:31.172420 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 22:05:31.172673 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":492,"threads_started":1}
I20260927 22:05:31.173861 19732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:31.273159 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 19609
I20260927 22:05:31.307153 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 19248
I20260927 22:05:31.308421 19310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:31.441540 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 19248
I20260927 22:05:51.470986 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38479
--webserver_interface=127.3.159.62
--webserver_port=41265
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:38479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:05:51.902774 19789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:51.903224 19789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:51.903335 19789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:51.911993 19789 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:05:51.912145 19789 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:05:51.912191 19789 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:05:51.912241 19789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:05:51.912312 19789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:51.912396 19789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:51.912438 19789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:05:51.912513 19789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:05:51.925480 19789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:38479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=41265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:51.927309 19789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:51.928958 19789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:51.961846 19795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:51.961956 19798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:51.961905 19796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:51.962247 19789 server_base.cc:1034] running on GCE node
I20260927 22:05:51.964293 19789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:51.966670 19789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:51.967998 19789 hybrid_clock.cc:648] HybridClock initialized: now 1790546751967901 us; error 89 us; skew 500 ppm
I20260927 22:05:51.968515 19789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:51 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546751, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:51.973529 19789 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:05:51.975916 19789 webserver.cc:460] Webserver started at http://127.3.159.62:41265/ using document root <none> and password file <none>
I20260927 22:05:51.976954 19789 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:51.977129 19789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:51.986249 19789 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 22:05:51.990926 19805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:51.993616 19789 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:05:51.994148 19789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "c075a36c4c7f4ab8b6182776a380861a"
format_stamp: "Formatted at 2026-09-27 22:05:27 on dist-test-slave-sg4j"
server_key: "9ce40c169c2bd22749375ae1deef931e"
server_key_iv: "b160d79dba4c3a28d922d06594e37ae4"
server_key_version: "encryptionkey@0"
I20260927 22:05:51.995914 19789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:52.025477 19789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:52.031646 19789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:52.032068 19789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:52.058590 19789 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:38479
I20260927 22:05:52.058646 19856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:38479 every 8 connection(s)
I20260927 22:05:52.060743 19789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:05:52.066831 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 19789
I20260927 22:05:52.067925 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:46743
--local_ip_for_outbound_sockets=127.3.159.1
--tserver_master_addrs=127.3.159.62:38479
--webserver_port=35173
--webserver_interface=127.3.159.1
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:05:52.070762 19857 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:05:52.078069 19857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Bootstrap starting.
I20260927 22:05:52.105032 19857 log.cc:824] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:52.120474 19857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:05:52.121590 19857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Bootstrap complete.
I20260927 22:05:52.133262 19857 raft_consensus.cc:348] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:05:52.134675 19857 raft_consensus.cc:729] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c075a36c4c7f4ab8b6182776a380861a, State: Initialized, Role: FOLLOWER
I20260927 22:05:52.135612 19857 consensus_queue.cc:260] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:05:52.135939 19857 raft_consensus.cc:388] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:05:52.136137 19857 raft_consensus.cc:482] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:05:52.136386 19857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 1 FOLLOWER]: Advancing to term 2
I20260927 22:05:52.141875 19857 raft_consensus.cc:504] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:05:52.142652 19857 leader_election.cc:302] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c075a36c4c7f4ab8b6182776a380861a; no voters:
I20260927 22:05:52.143642 19857 leader_election.cc:288] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 22:05:52.143953 19861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 2 FOLLOWER]: Leader election won for term 2
I20260927 22:05:52.145563 19861 raft_consensus.cc:686] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 2 LEADER]: Becoming Leader. State: Replica: c075a36c4c7f4ab8b6182776a380861a, State: Running, Role: LEADER
I20260927 22:05:52.146625 19857 sys_catalog.cc:564] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:05:52.146425 19861 consensus_queue.cc:237] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:05:52.153723 19862 sys_catalog.cc:455] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c075a36c4c7f4ab8b6182776a380861a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } } }
I20260927 22:05:52.154388 19862 sys_catalog.cc:458] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: This master's current role is: LEADER
I20260927 22:05:52.155622 19863 sys_catalog.cc:455] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c075a36c4c7f4ab8b6182776a380861a. Latest consensus state: current_term: 2 leader_uuid: "c075a36c4c7f4ab8b6182776a380861a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } } }
I20260927 22:05:52.156181 19863 sys_catalog.cc:458] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: This master's current role is: LEADER
I20260927 22:05:52.158864 19870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:05:52.170524 19870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:05:52.173297 19870 catalog_manager.cc:1259] Loaded cluster ID: 31d227c6f1f9408d8a917cbdb9af977b
I20260927 22:05:52.173629 19870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:05:52.178833 19870 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:05:52.181246 19870 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Loaded TSK: 0
I20260927 22:05:52.210615 19878 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:05:52.227942 19870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Generated new TSK 1
I20260927 22:05:52.228595 19870 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:05:52.549719 19859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:52.550179 19859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:52.550359 19859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:52.559280 19859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:52.559510 19859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:52.559734 19859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:05:52.572832 19859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:46743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=35173
--tserver_master_addrs=127.3.159.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:52.574817 19859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:52.576541 19859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:52.597045 19884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:52.597019 19885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:52.598059 19887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:52.604288 19859 server_base.cc:1034] running on GCE node
I20260927 22:05:52.605296 19859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:52.606910 19859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:52.608201 19859 hybrid_clock.cc:648] HybridClock initialized: now 1790546752608122 us; error 79 us; skew 500 ppm
I20260927 22:05:52.608696 19859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:52 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546752, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:52.613337 19859 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:05:52.615757 19859 webserver.cc:460] Webserver started at http://127.3.159.1:35173/ using document root <none> and password file <none>
I20260927 22:05:52.616815 19859 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:52.617054 19859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:52.624482 19859 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:05:52.628350 19894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:52.630330 19859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:05:52.630570 19859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "50331455b4084a198b2255e05bca12a8"
format_stamp: "Formatted at 2026-09-27 22:05:28 on dist-test-slave-sg4j"
server_key: "416e2165374e96a617ae65744c527bf1"
server_key_iv: "78931a352333ac43f77f8d4e7db0f28a"
server_key_version: "encryptionkey@0"
I20260927 22:05:52.631881 19859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:52.666730 19859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:52.673667 19859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:52.674072 19859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:52.675505 19859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:52.677618 19859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:52.677743 19859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:52.677891 19859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:52.677982 19859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:52.721951 19859 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:46743
I20260927 22:05:52.722016 20006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:46743 every 8 connection(s)
I20260927 22:05:52.723886 19859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:05:52.728861 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 19859
I20260927 22:05:52.729686 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:40879
--local_ip_for_outbound_sockets=127.3.159.2
--tserver_master_addrs=127.3.159.62:38479
--webserver_port=38603
--webserver_interface=127.3.159.2
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:05:52 dist-test-slave-sg4j krb5kdc[19235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546752, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:52.764460 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:52.727788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:49171 (local address 127.3.159.62:38479)
0927 22:05:52.728695 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:52.728718 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:52.729593 (+ 875us) server_negotiation.cc:407] Connection header received
0927 22:05:52.732649 (+ 3056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:52.732676 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:52.733268 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:52.733776 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:52.735710 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:52.737724 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:52.739648 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:52.740395 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:52.740650 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:52.752611 (+ 11961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:52.752624 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:52.752646 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:52.752707 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:52.757902 (+ 5195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:52.759135 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:52.759146 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:52.759153 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:52.759408 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:52.760040 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:52.760053 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:52.760055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:52.760739 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:52.761005 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:52.762213 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0927 22:05:52.762729 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":419,"threads_started":1}
I20260927 22:05:52.764513 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:52.728254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:49171)
0927 22:05:52.729299 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:52.729356 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:52.731856 (+ 2500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:52.733993 (+ 2137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:52.734003 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:52.734664 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:52.735398 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:52.735421 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:52.737933 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:52.737940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:52.739436 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:52.739453 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:52.740719 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:52.740727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:52.741014 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:52.747884 (+ 6870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:52.747926 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:52.752077 (+ 4151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:52.758218 (+ 6141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:52.758230 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:52.758248 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:52.758898 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:52.759607 (+ 709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:52.759612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:52.759615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:52.759877 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:52.760926 (+ 1049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:52.760937 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:52.761461 (+ 524us) client_negotiation.cc:769] Sending connection context
0927 22:05:52.761867 (+ 406us) client_negotiation.cc:240] Negotiation successful
0927 22:05:52.762703 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":431,"threads_started":1}
I20260927 22:05:52.768631 20007 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38479
I20260927 22:05:52.769153 20007 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:52.770268 20007 heartbeater.cc:511] Master 127.3.159.62:38479 requested a full tablet report, sending...
I20260927 22:05:52.773650 19822 ts_manager.cc:194] Registered new tserver with Master: 50331455b4084a198b2255e05bca12a8 (127.3.159.1:46743)
I20260927 22:05:52.775561 19822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:49171
W20260927 22:05:53.144780 20011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:53.145202 20011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:53.145380 20011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:53.153761 20011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:53.153920 20011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:53.154126 20011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:05:53.166045 20011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:40879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=38603
--tserver_master_addrs=127.3.159.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:53.167863 20011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:53.169677 20011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:53.189320 20017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:53.189661 20019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:53.190302 20011 server_base.cc:1034] running on GCE node
W20260927 22:05:53.190371 20016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:53.191334 20011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:53.192554 20011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:53.193904 20011 hybrid_clock.cc:648] HybridClock initialized: now 1790546753193798 us; error 64 us; skew 500 ppm
I20260927 22:05:53.194335 20011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:05:53 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546753, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:53.199226 20011 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:05:53.201591 20011 webserver.cc:460] Webserver started at http://127.3.159.2:38603/ using document root <none> and password file <none>
I20260927 22:05:53.202620 20011 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:53.202816 20011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:53.209836 20011 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:05:53.213519 20026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:53.215677 20011 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 22:05:53.215960 20011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "6399b7a7faab4109a65c59e36d7e19ed"
format_stamp: "Formatted at 2026-09-27 22:05:29 on dist-test-slave-sg4j"
server_key: "2bfb6747c4227d71e8d47648206b8f43"
server_key_iv: "5d1a74d4f07630f8c1182ec841b38c3b"
server_key_version: "encryptionkey@0"
I20260927 22:05:53.217260 20011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:53.241236 20011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:53.246188 20011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:53.246505 20011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:53.247956 20011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:53.249617 20011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:53.249748 20011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:53.249891 20011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:53.250001 20011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:53.292783 20011 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:40879
I20260927 22:05:53.292934 20138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:40879 every 8 connection(s)
I20260927 22:05:53.294919 20011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:05:53.301564 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 20011
I20260927 22:05:53.303025 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:36771
--local_ip_for_outbound_sockets=127.3.159.3
--tserver_master_addrs=127.3.159.62:38479
--webserver_port=46455
--webserver_interface=127.3.159.3
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 22:05:53 dist-test-slave-sg4j krb5kdc[19235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546753, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:53.323750 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:53.298417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:43063 (local address 127.3.159.62:38479)
0927 22:05:53.299004 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:53.299009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:53.300590 (+ 1581us) server_negotiation.cc:407] Connection header received
0927 22:05:53.303307 (+ 2717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:53.303318 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:53.303460 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:53.303643 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:53.305926 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.306753 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.308702 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.309211 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.309368 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:53.315194 (+ 5826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:53.315204 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:53.315208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:53.315273 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:53.318551 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:53.320099 (+ 1548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:53.320105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:53.320107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:53.320283 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:53.321016 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:53.321025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:53.321034 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:53.321463 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:53.321614 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:53.322964 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0927 22:05:53.323281 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":310,"threads_started":1}
I20260927 22:05:53.325477 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:53.299334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:43063)
0927 22:05:53.300349 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:53.300420 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:53.302818 (+ 2398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:53.304024 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:53.304036 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:53.304753 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:53.305653 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:53.305677 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.306975 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.306980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:53.308497 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:53.308514 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.309567 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.309576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:53.309871 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:53.310880 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:53.310917 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:53.314596 (+ 3679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:53.318831 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:53.318849 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:53.318883 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:53.319840 (+ 957us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:53.320435 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:53.320443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:53.320450 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:53.320789 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:53.321629 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:53.321641 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:53.322315 (+ 674us) client_negotiation.cc:769] Sending connection context
0927 22:05:53.322719 (+ 404us) client_negotiation.cc:240] Negotiation successful
0927 22:05:53.323506 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":446,"threads_started":1}
I20260927 22:05:53.327934 20139 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38479
I20260927 22:05:53.328485 20139 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:53.329877 20139 heartbeater.cc:511] Master 127.3.159.62:38479 requested a full tablet report, sending...
I20260927 22:05:53.332209 19822 ts_manager.cc:194] Registered new tserver with Master: 6399b7a7faab4109a65c59e36d7e19ed (127.3.159.2:40879)
I20260927 22:05:53.333159 19822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:43063
W20260927 22:05:53.728557 20143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:05:53.729058 20143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:05:53.729228 20143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:05:53.738487 20143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:05:53.738655 20143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:05:53.738795 20143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:05:53.750942 20143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:36771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=46455
--tserver_master_addrs=127.3.159.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:05:53.752792 20143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:05:53.754595 20143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:05:53.773133 20148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:53.773139 20151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:05:53.773686 20149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:05:53.775921 20143 server_base.cc:1034] running on GCE node
I20260927 22:05:53.776855 20143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:05:53.778200 20143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:05:53.779574 20143 hybrid_clock.cc:648] HybridClock initialized: now 1790546753779473 us; error 97 us; skew 500 ppm
I20260927 22:05:53.780027 20143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:05:53.780702 20007 heartbeater.cc:503] Master 127.3.159.62:38479 was elected leader, sending a full tablet report...
Sep 27 22:05:53 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546753, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:05:53.784307 20143 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:05:53.786761 20143 webserver.cc:460] Webserver started at http://127.3.159.3:46455/ using document root <none> and password file <none>
I20260927 22:05:53.788475 20143 fs_manager.cc:359] Metadata directory not provided
I20260927 22:05:53.788779 20143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:05:53.795887 20143 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:05:53.799942 20158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:53.802280 20143 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 22:05:53.802536 20143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "32926f9156864f04a58b93c4b6b9078a"
format_stamp: "Formatted at 2026-09-27 22:05:30 on dist-test-slave-sg4j"
server_key: "e6402182a75ee53b28fd349d3ab3ae1d"
server_key_iv: "8705159ac96266c07818456c3d6dd096"
server_key_version: "encryptionkey@0"
I20260927 22:05:53.804373 20143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:05:53.833307 20143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:05:53.838789 20143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:05:53.839128 20143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:05:53.840960 20143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:05:53.842834 20143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:05:53.842990 20143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:53.843150 20143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:05:53.843243 20143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:05:53.888234 20143 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:36771
I20260927 22:05:53.888389 20270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:36771 every 8 connection(s)
I20260927 22:05:53.890293 20143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:05:53.892396 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 20143
Sep 27 22:05:53 dist-test-slave-sg4j krb5kdc[19235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546753, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:05:53.927124 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:53.894236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.3.159.62:38479)
0927 22:05:53.906620 (+ 12384us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:53.906631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:53.907740 (+ 1109us) server_negotiation.cc:407] Connection header received
0927 22:05:53.908062 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:53.908072 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:53.908284 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:53.908523 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:53.911075 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.912106 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.914328 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.914870 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.914994 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:53.919346 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:53.919352 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:53.919357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:53.919409 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:53.923044 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:53.924390 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:53.924398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:53.924401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:53.924594 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:53.925113 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:53.925122 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:53.925126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:53.925606 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:53.925961 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:53.926379 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 22:05:53.926682 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12136,"thread_start_us":12388,"threads_started":1}
I20260927 22:05:53.934207 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:53.894692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:51625 (local address 127.3.159.62:38479)
0927 22:05:53.910027 (+ 15335us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:53.910032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:53.910051 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:05:53.912938 (+ 2887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:53.912943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:53.913045 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:53.913178 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:53.915517 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.916702 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.919637 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.920260 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.920438 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:53.925550 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:53.925562 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:53.925566 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:53.925620 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:53.929685 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:53.931124 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:53.931132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:53.931137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:53.931356 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:53.931936 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:53.931941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:53.931943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:53.932397 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:53.932548 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:53.933589 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0927 22:05:53.933843 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15180,"thread_start_us":14972,"threads_started":1}
I20260927 22:05:53.935421 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:53.895755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:51625)
0927 22:05:53.909818 (+ 14063us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:53.909925 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:53.912542 (+ 2617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:53.913600 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:53.913615 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:53.914460 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:05:53.915281 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:53.915312 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.916900 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.916910 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:53.919420 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:53.919436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.920546 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.920554 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:53.920847 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:53.921713 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:05:53.921740 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:05:53.925051 (+ 3311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:05:53.929914 (+ 4863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:53.929928 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:53.929950 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:53.930848 (+ 898us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:53.931507 (+ 659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:05:53.931511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:05:53.931514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:05:53.931765 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:05:53.932570 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:05:53.932577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:05:53.932997 (+ 420us) client_negotiation.cc:769] Sending connection context
0927 22:05:53.933279 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 22:05:53.933864 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12627,"thread_start_us":12283,"threads_started":1}
I20260927 22:05:53.937784 20271 heartbeater.cc:347] Connected to a master server at 127.3.159.62:38479
I20260927 22:05:53.938261 20271 heartbeater.cc:464] Registering TS with master...
I20260927 22:05:53.939127 20271 heartbeater.cc:511] Master 127.3.159.62:38479 requested a full tablet report, sending...
I20260927 22:05:53.941143 19822 ts_manager.cc:194] Registered new tserver with Master: 32926f9156864f04a58b93c4b6b9078a (127.3.159.3:36771)
I20260927 22:05:53.942104 19822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:51625
I20260927 22:05:53.945277 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:05:53.954843 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:53.946830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.3.159.62:38479)
0927 22:05:53.947180 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:53.947185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:53.947634 (+ 449us) server_negotiation.cc:407] Connection header received
0927 22:05:53.947985 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:53.947990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:53.948225 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:53.948408 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:05:53.949035 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.950018 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.951950 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.952483 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.952861 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:53.953762 (+ 901us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:05:53.954095 (+ 333us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 22:05:53.954515 (+ 420us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48366: token expired
Metrics: {"server-negotiator.queue_time_us":106}
W20260927 22:05:53.955051 20275 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48366: token expired
W20260927 22:05:53.955757 19744 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.159.62:38479)
I20260927 22:05:53.973109 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:53.956458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.3.159.62:38479)
0927 22:05:53.956732 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:53.956739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:53.956932 (+ 193us) server_negotiation.cc:407] Connection header received
0927 22:05:53.957182 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:53.957188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:53.957273 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:53.957437 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:53.958996 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.959817 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.961370 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:53.961840 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:53.962011 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:53.965755 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:53.965765 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:53.965769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:53.965886 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:53.969819 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:53.970794 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:53.970803 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:53.970811 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:53.971028 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:53.971445 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:53.971453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:53.971459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:53.971829 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:53.972162 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:53.972480 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 22:05:53.972744 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 22:05:53.978803 19821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:05:53.987864 19821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:05:54.024583 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.006403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:40879 (local address 127.0.0.1:55060)
0927 22:05:54.007624 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.007687 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:54.008124 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:54.011062 (+ 2938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:54.011072 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:54.011108 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:54.012115 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:54.012138 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.017016 (+ 4878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.017022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:54.020541 (+ 3519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:54.020562 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.023076 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.023084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:54.023403 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.023461 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 22:05:54.023661 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 22:05:54.023927 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":722,"threads_started":1}
I20260927 22:05:54.025194 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.006572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.3.159.2:40879)
0927 22:05:54.010509 (+ 3937us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:54.010518 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:54.010540 (+ 22us) server_negotiation.cc:407] Connection header received
0927 22:05:54.010633 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:54.010640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:54.010813 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:54.011666 (+ 853us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:54.013234 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.016509 (+ 3275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.020821 (+ 4312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.022635 (+ 1814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.022989 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.023969 (+ 980us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:54.024066 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 22:05:54.024397 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3742,"thread_start_us":3566,"threads_started":1}
I20260927 22:05:54.026453 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.006843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:36771 (local address 127.0.0.1:46484)
0927 22:05:54.010000 (+ 3157us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.010023 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:54.010195 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:54.011369 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:54.011373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:54.011394 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:54.012478 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:54.012486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.015520 (+ 3034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.016044 (+ 524us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:54.021276 (+ 5232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:54.021309 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.025347 (+ 4038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.025354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:54.025600 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.025642 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 22:05:54.025753 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:05:54.025947 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3001,"mutex_wait_us":182,"thread_start_us":2792,"threads_started":1}
I20260927 22:05:54.027266 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.010376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46743 (local address 127.0.0.1:60074)
0927 22:05:54.013082 (+ 2706us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.013104 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:54.013241 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:54.015552 (+ 2311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:54.015559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:54.015598 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:54.016608 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:54.016631 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.019404 (+ 2773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.019411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:54.021916 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:54.021936 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.026421 (+ 4485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.026431 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:54.026628 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.026665 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:05:54.026756 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 22:05:54.026910 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2518,"mutex_wait_us":77,"thread_start_us":2343,"threads_started":1}
I20260927 22:05:54.027447 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.007449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.3.159.3:36771)
0927 22:05:54.009453 (+ 2004us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:54.009466 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:54.010345 (+ 879us) server_negotiation.cc:407] Connection header received
0927 22:05:54.010466 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:54.010476 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:54.010778 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:54.011008 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:54.012686 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.015338 (+ 2652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.021931 (+ 6593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.024718 (+ 2787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.024884 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.025160 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:54.026857 (+ 1697us) server_negotiation.cc:299] Negotiation successful
0927 22:05:54.027062 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1695,"thread_start_us":1377,"threads_started":1}
I20260927 22:05:54.028100 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.009262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.3.159.1:46743)
0927 22:05:54.012227 (+ 2965us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:54.012238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:54.013726 (+ 1488us) server_negotiation.cc:407] Connection header received
0927 22:05:54.013897 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:54.013907 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:54.014492 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:54.015110 (+ 618us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:54.017101 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.019010 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.022171 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.025854 (+ 3683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.025989 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.026227 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:54.027566 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0927 22:05:54.027776 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2732,"thread_start_us":2561,"threads_started":1}
I20260927 22:05:54.031560 20074 tablet_service.cc:1467] Processing CreateTablet for tablet 3701666652f54e5caa0f32855e898f8b (DEFAULT_TABLE table=test-table [id=12c802012a694d158d8fb19f54a48cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:54.034468 20074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3701666652f54e5caa0f32855e898f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:54.033607 19942 tablet_service.cc:1467] Processing CreateTablet for tablet 3701666652f54e5caa0f32855e898f8b (DEFAULT_TABLE table=test-table [id=12c802012a694d158d8fb19f54a48cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:54.033258 20206 tablet_service.cc:1467] Processing CreateTablet for tablet 3701666652f54e5caa0f32855e898f8b (DEFAULT_TABLE table=test-table [id=12c802012a694d158d8fb19f54a48cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:05:54.035682 19942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3701666652f54e5caa0f32855e898f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:54.036072 20206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3701666652f54e5caa0f32855e898f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:05:54.055373 20288 tablet_bootstrap.cc:492] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: Bootstrap starting.
I20260927 22:05:54.058449 20290 tablet_bootstrap.cc:492] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: Bootstrap starting.
I20260927 22:05:54.065037 20288 tablet_bootstrap.cc:654] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:54.065768 20290 tablet_bootstrap.cc:654] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:54.067320 20289 tablet_bootstrap.cc:492] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: Bootstrap starting.
I20260927 22:05:54.068754 20290 log.cc:824] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:54.068758 20288 log.cc:824] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:54.076227 20289 tablet_bootstrap.cc:654] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: Neither blocks nor log segments found. Creating new log.
I20260927 22:05:54.078310 20289 log.cc:824] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: Log is configured to *not* fsync() on all Append() calls
I20260927 22:05:54.083468 20290 tablet_bootstrap.cc:492] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: No bootstrap required, opened a new log
I20260927 22:05:54.083609 20288 tablet_bootstrap.cc:492] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: No bootstrap required, opened a new log
I20260927 22:05:54.084061 20290 ts_tablet_manager.cc:1397] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.013s
I20260927 22:05:54.087055 20288 ts_tablet_manager.cc:1397] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.006s
I20260927 22:05:54.089018 20289 tablet_bootstrap.cc:492] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: No bootstrap required, opened a new log
I20260927 22:05:54.089556 20289 ts_tablet_manager.cc:1397] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.013s
I20260927 22:05:54.099236 20289 raft_consensus.cc:348] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } } peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } }
I20260927 22:05:54.099879 20289 raft_consensus.cc:374] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:54.100026 20289 raft_consensus.cc:729] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6399b7a7faab4109a65c59e36d7e19ed, State: Initialized, Role: FOLLOWER
I20260927 22:05:54.100507 20290 raft_consensus.cc:348] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } } peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } }
I20260927 22:05:54.101480 20290 raft_consensus.cc:374] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:54.101320 20289 consensus_queue.cc:260] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } } peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } }
I20260927 22:05:54.101657 20290 raft_consensus.cc:729] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50331455b4084a198b2255e05bca12a8, State: Initialized, Role: FOLLOWER
I20260927 22:05:54.102523 20290 consensus_queue.cc:260] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } } peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } }
I20260927 22:05:54.102648 20288 raft_consensus.cc:348] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } } peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } }
I20260927 22:05:54.103161 20288 raft_consensus.cc:374] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:05:54.103315 20288 raft_consensus.cc:729] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32926f9156864f04a58b93c4b6b9078a, State: Initialized, Role: FOLLOWER
I20260927 22:05:54.104257 20288 consensus_queue.cc:260] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } } peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } }
I20260927 22:05:54.105225 20139 heartbeater.cc:503] Master 127.3.159.62:38479 was elected leader, sending a full tablet report...
I20260927 22:05:54.106271 20271 heartbeater.cc:503] Master 127.3.159.62:38479 was elected leader, sending a full tablet report...
I20260927 22:05:54.106634 20289 ts_tablet_manager.cc:1428] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20260927 22:05:54.107589 20288 ts_tablet_manager.cc:1428] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: Time spent starting tablet: real 0.020s user 0.017s sys 0.001s
I20260927 22:05:54.110517 20290 ts_tablet_manager.cc:1428] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: Time spent starting tablet: real 0.026s user 0.018s sys 0.004s
W20260927 22:05:54.143095 20272 tablet.cc:2369] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:05:54.216078 20295 raft_consensus.cc:482] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:05:54.216473 20295 raft_consensus.cc:504] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } } peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } }
I20260927 22:05:54.218880 20295 leader_election.cc:288] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6399b7a7faab4109a65c59e36d7e19ed (127.3.159.2:40879), 32926f9156864f04a58b93c4b6b9078a (127.3.159.3:36771)
W20260927 22:05:54.227399 20008 tablet.cc:2369] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:05:54.229583 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.219566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:59517 (local address 127.3.159.3:36771)
0927 22:05:54.219788 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:54.219795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:54.221417 (+ 1622us) server_negotiation.cc:407] Connection header received
0927 22:05:54.221690 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:54.221696 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:54.221788 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:54.221889 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:54.222956 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.224180 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.227144 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.228202 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.228385 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.228526 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:54.229115 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 22:05:54.229249 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 22:05:54.229857 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.219349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:36771 (local address 127.3.159.1:59517)
0927 22:05:54.221133 (+ 1784us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.221164 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:54.221472 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:54.221984 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:54.221989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:54.222024 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:54.222567 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:54.222585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.224514 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.224520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:54.226829 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:54.226856 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.228493 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.228500 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:54.228706 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.228757 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:05:54.229074 (+ 317us) client_negotiation.cc:240] Negotiation successful
0927 22:05:54.229268 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1317,"mutex_wait_us":4,"thread_start_us":1005,"threads_started":1}
I20260927 22:05:54.230078 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.219707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:40441 (local address 127.3.159.2:40879)
0927 22:05:54.220005 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:54.220017 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:54.221570 (+ 1553us) server_negotiation.cc:407] Connection header received
0927 22:05:54.221975 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:54.221981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:54.222082 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:54.222224 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:05:54.223450 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.224385 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.227257 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.228547 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.228726 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.228856 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:54.229536 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 22:05:54.229803 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 22:05:54.230222 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.219458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:40879 (local address 127.3.159.1:40441)
0927 22:05:54.221405 (+ 1947us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.221437 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:05:54.221668 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:05:54.222389 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:05:54.222394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:05:54.222425 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:05:54.223177 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:54.223194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.224603 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.224608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:54.226939 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:05:54.226955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.229021 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.229034 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:05:54.229274 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.229317 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:05:54.229507 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 22:05:54.229677 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1703,"mutex_wait_us":87,"thread_start_us":1106,"threads_started":1}
I20260927 22:05:54.231317 20226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3701666652f54e5caa0f32855e898f8b" candidate_uuid: "50331455b4084a198b2255e05bca12a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32926f9156864f04a58b93c4b6b9078a" is_pre_election: true
I20260927 22:05:54.231732 20094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3701666652f54e5caa0f32855e898f8b" candidate_uuid: "50331455b4084a198b2255e05bca12a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6399b7a7faab4109a65c59e36d7e19ed" is_pre_election: true
I20260927 22:05:54.231997 20226 raft_consensus.cc:2454] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50331455b4084a198b2255e05bca12a8 in term 0.
I20260927 22:05:54.232338 20094 raft_consensus.cc:2454] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50331455b4084a198b2255e05bca12a8 in term 0.
I20260927 22:05:54.232990 19896 leader_election.cc:302] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32926f9156864f04a58b93c4b6b9078a, 50331455b4084a198b2255e05bca12a8; no voters:
I20260927 22:05:54.233626 20295 raft_consensus.cc:2781] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:05:54.233846 20295 raft_consensus.cc:482] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:05:54.233969 20295 raft_consensus.cc:3037] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:54.236711 20295 raft_consensus.cc:504] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } } peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } }
I20260927 22:05:54.237592 20295 leader_election.cc:288] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [CANDIDATE]: Term 1 election: Requested vote from peers 6399b7a7faab4109a65c59e36d7e19ed (127.3.159.2:40879), 32926f9156864f04a58b93c4b6b9078a (127.3.159.3:36771)
I20260927 22:05:54.238507 20226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3701666652f54e5caa0f32855e898f8b" candidate_uuid: "50331455b4084a198b2255e05bca12a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32926f9156864f04a58b93c4b6b9078a"
I20260927 22:05:54.238629 20094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3701666652f54e5caa0f32855e898f8b" candidate_uuid: "50331455b4084a198b2255e05bca12a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6399b7a7faab4109a65c59e36d7e19ed"
I20260927 22:05:54.238881 20226 raft_consensus.cc:3037] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:54.238952 20094 raft_consensus.cc:3037] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:05:54.241869 20226 raft_consensus.cc:2454] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50331455b4084a198b2255e05bca12a8 in term 1.
I20260927 22:05:54.242477 20094 raft_consensus.cc:2454] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50331455b4084a198b2255e05bca12a8 in term 1.
I20260927 22:05:54.242702 19896 leader_election.cc:302] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32926f9156864f04a58b93c4b6b9078a, 50331455b4084a198b2255e05bca12a8; no voters:
I20260927 22:05:54.243338 20295 raft_consensus.cc:2781] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:05:54.244107 20295 raft_consensus.cc:686] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 1 LEADER]: Becoming Leader. State: Replica: 50331455b4084a198b2255e05bca12a8, State: Running, Role: LEADER
I20260927 22:05:54.244854 20295 consensus_queue.cc:237] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } } peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } }
I20260927 22:05:54.251389 19822 catalog_manager.cc:5495] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50331455b4084a198b2255e05bca12a8 (127.3.159.1). New cstate: current_term: 1 leader_uuid: "50331455b4084a198b2255e05bca12a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50331455b4084a198b2255e05bca12a8" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 } health_report { overall_health: UNKNOWN } } }
I20260927 22:05:54.286914 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.279069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.3.159.1:46743)
0927 22:05:54.279367 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:54.279374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:54.279400 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:05:54.279533 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:54.279545 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:54.279725 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:54.279915 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:05:54.280432 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.281458 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.283645 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.284225 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.284377 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.285291 (+ 914us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:05:54.286078 (+ 787us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:05:54.286236 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:54.286470 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 22:05:54.286577 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260927 22:05:54.297498 20140 tablet.cc:2369] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:05:54.299117 20226 raft_consensus.cc:1261] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a [term 1 FOLLOWER]: Refusing update from remote peer 50331455b4084a198b2255e05bca12a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:05:54.299171 20094 raft_consensus.cc:1261] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed [term 1 FOLLOWER]: Refusing update from remote peer 50331455b4084a198b2255e05bca12a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:05:54.300755 20295 consensus_queue.cc:1035] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32926f9156864f04a58b93c4b6b9078a" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:05:54.301366 20299 consensus_queue.cc:1035] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6399b7a7faab4109a65c59e36d7e19ed" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 22:05:54.340421 19821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:05:54.340809 19821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:05:54.345839 19821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Sending DeleteTablet for 3 replicas of tablet 3701666652f54e5caa0f32855e898f8b
I20260927 22:05:54.347621 20206 tablet_service.cc:1514] Processing DeleteTablet for tablet 3701666652f54e5caa0f32855e898f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:54 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:46484
I20260927 22:05:54.347862 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 19859
I20260927 22:05:54.347793 20074 tablet_service.cc:1514] Processing DeleteTablet for tablet 3701666652f54e5caa0f32855e898f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:54 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:55060
I20260927 22:05:54.349241 19942 tablet_service.cc:1514] Processing DeleteTablet for tablet 3701666652f54e5caa0f32855e898f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:05:54 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:60074
I20260927 22:05:54.350337 20312 tablet_replica.cc:330] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: stopping tablet replica
I20260927 22:05:54.353837 20312 raft_consensus.cc:2229] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:05:54.358036 20312 raft_consensus.cc:2258] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:54.362732 20313 tablet_replica.cc:330] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: stopping tablet replica
I20260927 22:05:54.364223 20313 raft_consensus.cc:2229] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:05:54.365058 20311 tablet_replica.cc:330] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: stopping tablet replica
I20260927 22:05:54.366217 20313 raft_consensus.cc:2258] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:54.367027 20311 raft_consensus.cc:2229] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:05:54.367726 20311 raft_consensus.cc:2258] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:05:54.371106 20312 ts_tablet_manager.cc:1905] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:54.371691 20313 ts_tablet_manager.cc:1905] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:54.379972 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.348891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.3.159.1:46743)
0927 22:05:54.350225 (+ 1334us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:54.350232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:54.350258 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:05:54.350333 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:54.350338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:54.350433 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:54.350627 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:54.353093 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.354597 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.357797 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.358589 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.358825 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.366102 (+ 7277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:54.366267 (+ 165us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:54.366337 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:54.366467 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:54.373110 (+ 6643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:54.374848 (+ 1738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:54.374863 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:54.374870 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:54.375108 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:54.375612 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:54.375616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:54.375619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:54.376383 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:54.376849 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:54.379093 (+ 2244us) server_negotiation.cc:299] Negotiation successful
0927 22:05:54.379486 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1197}
I20260927 22:05:54.381206 20312 ts_tablet_manager.cc:1918] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:54.381621 20312 log.cc:1196] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/3701666652f54e5caa0f32855e898f8b
I20260927 22:05:54.382654 20002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:54.382786 20312 ts_tablet_manager.cc:1939] T 3701666652f54e5caa0f32855e898f8b P 6399b7a7faab4109a65c59e36d7e19ed: Deleting consensus metadata
I20260927 22:05:54.381635 20313 ts_tablet_manager.cc:1918] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:54.383496 20313 log.cc:1196] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/3701666652f54e5caa0f32855e898f8b
I20260927 22:05:54.384440 20311 ts_tablet_manager.cc:1905] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:05:54.386150 19808 catalog_manager.cc:4841] TS 6399b7a7faab4109a65c59e36d7e19ed (127.3.159.2:40879): tablet 3701666652f54e5caa0f32855e898f8b (table test-table [id=12c802012a694d158d8fb19f54a48cd3]) successfully deleted
I20260927 22:05:54.394333 20311 ts_tablet_manager.cc:1918] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:05:54.394769 20311 log.cc:1196] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/3701666652f54e5caa0f32855e898f8b
I20260927 22:05:54.395357 20311 ts_tablet_manager.cc:1939] T 3701666652f54e5caa0f32855e898f8b P 32926f9156864f04a58b93c4b6b9078a: Deleting consensus metadata
I20260927 22:05:54.398285 19807 catalog_manager.cc:4841] TS 32926f9156864f04a58b93c4b6b9078a (127.3.159.3:36771): tablet 3701666652f54e5caa0f32855e898f8b (table test-table [id=12c802012a694d158d8fb19f54a48cd3]) successfully deleted
I20260927 22:05:54.508697 20313 ts_tablet_manager.cc:1939] T 3701666652f54e5caa0f32855e898f8b P 50331455b4084a198b2255e05bca12a8: Deleting consensus metadata
I20260927 22:05:54.509748 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 19859
I20260927 22:05:54.535560 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 20011
W20260927 22:05:54.535780 19808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.159.1:46743 (error 108)
I20260927 22:05:54.538393 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.537293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46743 (local address 127.0.0.1:60114)
0927 22:05:54.537965 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.538169 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":346,"threads_started":1}
W20260927 22:05:54.538782 19808 catalog_manager.cc:4584] TS 50331455b4084a198b2255e05bca12a8 (127.3.159.1:46743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3701666652f54e5caa0f32855e898f8b: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46743: connect: Connection refused (error 111)
I20260927 22:05:54.554749 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.536753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55064 (local address 127.3.159.2:40879)
0927 22:05:54.537434 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:54.537453 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:54.537501 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:05:54.537600 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:54.537608 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:54.537726 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:54.537867 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:54.539997 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.540969 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.542755 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.543191 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.543359 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.546715 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:54.546740 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:54.546763 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:54.546820 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:54.551043 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:54.552114 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:54.552124 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:54.552127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:54.552305 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:54.552759 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:54.552765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:54.552770 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:54.553272 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:54.553681 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:54.554032 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 22:05:54.554325 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499}
I20260927 22:05:54.555558 20134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:54.593019 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.592175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46743 (local address 127.0.0.1:60126)
0927 22:05:54.592584 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.592767 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260927 22:05:54.594841 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.594232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46743 (local address 127.0.0.1:60142)
0927 22:05:54.594481 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.594608 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 22:05:54.656416 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.655434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46743 (local address 127.0.0.1:60144)
0927 22:05:54.656039 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.656198 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
I20260927 22:05:54.658169 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.657662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46743 (local address 127.0.0.1:60148)
0927 22:05:54.657883 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.657989 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 22:05:54.662433 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 20011
I20260927 22:05:54.686044 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 20143
I20260927 22:05:54.705521 20285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.687198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.3.159.3:36771)
0927 22:05:54.687538 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 22:05:54.687543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:05:54.687567 (+ 24us) server_negotiation.cc:407] Connection header received
0927 22:05:54.687639 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:05:54.687643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:05:54.687733 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:05:54.687876 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:05:54.689660 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.690497 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.692462 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:05:54.692985 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:05:54.693148 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:05:54.696642 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:05:54.696661 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:05:54.696679 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:05:54.696745 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:05:54.700465 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:54.702071 (+ 1606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:54.702081 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:54.702083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:54.702491 (+ 408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:05:54.703261 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:05:54.703266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:05:54.703268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:05:54.703746 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:05:54.704204 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0927 22:05:54.704789 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 22:05:54.705063 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260927 22:05:54.706378 20266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:54.739669 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.738996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46743 (local address 127.0.0.1:60156)
0927 22:05:54.739324 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.739455 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260927 22:05:54.741653 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.740941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46743 (local address 127.0.0.1:60164)
0927 22:05:54.741235 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.741353 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260927 22:05:54.816557 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 20143
I20260927 22:05:54.837188 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 19789
I20260927 22:05:54.838929 19852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:05:54.982070 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:05:54.981178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46743 (local address 127.0.0.1:60174)
0927 22:05:54.981658 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0927 22:05:54.981794 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289}
I20260927 22:05:54.982678 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 19789
I20260927 22:06:15.010737 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38479
--webserver_interface=127.3.159.62
--webserver_port=41265
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:38479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:06:15.455090 20320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:15.455523 20320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:15.455633 20320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:15.464291 20320 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 22:06:15.464434 20320 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 22:06:15.464478 20320 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 22:06:15.464536 20320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:06:15.464591 20320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:15.464670 20320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:15.464721 20320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:06:15.464777 20320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:06:15.476776 20320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:38479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:38479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=41265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:15.478574 20320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:15.480188 20320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:15.495628 20325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:15.495801 20328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:15.495843 20326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:15.496759 20320 server_base.cc:1034] running on GCE node
I20260927 22:06:15.498052 20320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:15.500255 20320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:15.501755 20320 hybrid_clock.cc:648] HybridClock initialized: now 1790546775501626 us; error 89 us; skew 500 ppm
I20260927 22:06:15.502269 20320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:15 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546775, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:15.511641 20320 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:06:15.514228 20320 webserver.cc:460] Webserver started at http://127.3.159.62:41265/ using document root <none> and password file <none>
I20260927 22:06:15.515311 20320 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:15.515509 20320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:15.524194 20320 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.003s
I20260927 22:06:15.529122 20335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:15.531376 20320 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 22:06:15.531710 20320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "c075a36c4c7f4ab8b6182776a380861a"
format_stamp: "Formatted at 2026-09-27 22:05:27 on dist-test-slave-sg4j"
server_key: "9ce40c169c2bd22749375ae1deef931e"
server_key_iv: "b160d79dba4c3a28d922d06594e37ae4"
server_key_version: "encryptionkey@0"
I20260927 22:06:15.533217 20320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:15.560581 20320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:15.567313 20320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:15.567664 20320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:15.593954 20320 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:38479
I20260927 22:06:15.594025 20386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:38479 every 8 connection(s)
I20260927 22:06:15.595983 20320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:06:15.598308 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 20320
I20260927 22:06:15.599505 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:46743
--local_ip_for_outbound_sockets=127.3.159.1
--tserver_master_addrs=127.3.159.62:38479
--webserver_port=35173
--webserver_interface=127.3.159.1
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:06:15.605302 20387 sys_catalog.cc:263] Verifying existing consensus state
I20260927 22:06:15.610499 20387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Bootstrap starting.
I20260927 22:06:15.645955 20387 log.cc:824] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:15.676553 20387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 22:06:15.677338 20387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Bootstrap complete.
I20260927 22:06:15.697721 20387 raft_consensus.cc:348] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:06:15.699128 20387 raft_consensus.cc:729] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c075a36c4c7f4ab8b6182776a380861a, State: Initialized, Role: FOLLOWER
I20260927 22:06:15.700003 20387 consensus_queue.cc:260] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:06:15.700330 20387 raft_consensus.cc:388] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:06:15.700469 20387 raft_consensus.cc:482] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:06:15.700665 20387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 2 FOLLOWER]: Advancing to term 3
I20260927 22:06:15.706732 20387 raft_consensus.cc:504] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:06:15.707535 20387 leader_election.cc:302] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c075a36c4c7f4ab8b6182776a380861a; no voters:
I20260927 22:06:15.708385 20387 leader_election.cc:288] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 22:06:15.708742 20391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 3 FOLLOWER]: Leader election won for term 3
I20260927 22:06:15.710450 20391 raft_consensus.cc:686] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [term 3 LEADER]: Becoming Leader. State: Replica: c075a36c4c7f4ab8b6182776a380861a, State: Running, Role: LEADER
I20260927 22:06:15.711109 20387 sys_catalog.cc:564] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:06:15.711329 20391 consensus_queue.cc:237] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } }
I20260927 22:06:15.716557 20392 sys_catalog.cc:455] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c075a36c4c7f4ab8b6182776a380861a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } } }
I20260927 22:06:15.716451 20393 sys_catalog.cc:455] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c075a36c4c7f4ab8b6182776a380861a. Latest consensus state: current_term: 3 leader_uuid: "c075a36c4c7f4ab8b6182776a380861a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c075a36c4c7f4ab8b6182776a380861a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 38479 } } }
I20260927 22:06:15.716979 20393 sys_catalog.cc:458] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:15.716981 20392 sys_catalog.cc:458] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:15.718330 20400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:06:15.734527 20400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:06:15.737026 20400 catalog_manager.cc:1259] Loaded cluster ID: 31d227c6f1f9408d8a917cbdb9af977b
I20260927 22:06:15.737313 20400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:06:15.743065 20400 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:06:15.745611 20400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Loaded TSK: 0
I20260927 22:06:15.745970 20400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c075a36c4c7f4ab8b6182776a380861a: Loaded TSK: 1
F20260927 22:06:15.747607 20400 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f41c4e969ad google::LogMessage::Fail() at ??:0
@ 0x7f41c4e9988a google::LogMessage::SendToLog() at ??:0
@ 0x7f41c4e964de google::LogMessage::Flush() at ??:0
@ 0x7f41c4e97a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f41c8bfa7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f41c8be973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f41da8c91c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f41da8c8783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f41da9542a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f41da9540c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f41da99e5e9 std::function<>::operator()() at ??:0
@ 0x7f41da8cccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f41da8cac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f41da952610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f41da952439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f41c7ae2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f41c7ae591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f41c7ae5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f41c7ac0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f41cad9a6db start_thread at ??:0
@ 0x7f41c3d3061f clone at ??:0
W20260927 22:06:16.121889 20389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:16.122215 20389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:16.122315 20389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:16.130455 20389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:16.130609 20389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:16.130733 20389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:06:16.142239 20389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:46743
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=35173
--tserver_master_addrs=127.3.159.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:16.143981 20389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:16.145618 20389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:16.163985 20437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:16.164084 20435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:16.164070 20434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:16.165047 20389 server_base.cc:1034] running on GCE node
I20260927 22:06:16.166057 20389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:16.167454 20389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:16.168764 20389 hybrid_clock.cc:648] HybridClock initialized: now 1790546776168715 us; error 67 us; skew 500 ppm
I20260927 22:06:16.169240 20389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:16 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:16.174058 20389 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:06:16.176301 20389 webserver.cc:460] Webserver started at http://127.3.159.1:35173/ using document root <none> and password file <none>
I20260927 22:06:16.177337 20389 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:16.177618 20389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:16.186123 20389 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 22:06:16.191200 20444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:16.193442 20389 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:06:16.193810 20389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "50331455b4084a198b2255e05bca12a8"
format_stamp: "Formatted at 2026-09-27 22:05:28 on dist-test-slave-sg4j"
server_key: "416e2165374e96a617ae65744c527bf1"
server_key_iv: "78931a352333ac43f77f8d4e7db0f28a"
server_key_version: "encryptionkey@0"
I20260927 22:06:16.195221 20389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:16.221341 20389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:16.227710 20389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:16.228113 20389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:16.229620 20389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:16.231462 20389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:16.231617 20389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:16.231791 20389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:16.231887 20389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:16.276058 20389 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:46743
I20260927 22:06:16.276142 20556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:46743 every 8 connection(s)
I20260927 22:06:16.278021 20389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:06:16.280365 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 20389
I20260927 22:06:16.281118 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:40879
--local_ip_for_outbound_sockets=127.3.159.2
--tserver_master_addrs=127.3.159.62:38479
--webserver_port=38603
--webserver_interface=127.3.159.2
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:06:16.285468 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:16.282094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:60891)
0927 22:06:16.283362 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:16.283775 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":533,"threads_started":1}
W20260927 22:06:16.287212 20557 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:38479 (0 consecutive failures): Network error: Failed to ping master at 127.3.159.62:38479: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
I20260927 22:06:16.289309 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:16.288706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:60229)
0927 22:06:16.289020 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:16.289137 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 22:06:16.290686 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:16.290285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:45147)
0927 22:06:16.290449 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:16.290555 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 22:06:16.291000 20557 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:06:16.696408 20560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:16.696866 20560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:16.697109 20560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:16.705768 20560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:16.705928 20560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:16.706127 20560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:06:16.718036 20560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:40879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=38603
--tserver_master_addrs=127.3.159.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:16.719900 20560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:16.721702 20560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:16.738747 20568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:16.738870 20566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:16.738839 20565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:16.740414 20560 server_base.cc:1034] running on GCE node
I20260927 22:06:16.741366 20560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:16.743001 20560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:16.744351 20560 hybrid_clock.cc:648] HybridClock initialized: now 1790546776744276 us; error 59 us; skew 500 ppm
I20260927 22:06:16.744920 20560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:16 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:16.749750 20560 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:06:16.752187 20560 webserver.cc:460] Webserver started at http://127.3.159.2:38603/ using document root <none> and password file <none>
I20260927 22:06:16.753219 20560 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:16.753475 20560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:16.760959 20560 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 22:06:16.766806 20575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:16.768931 20560 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:06:16.769248 20560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "6399b7a7faab4109a65c59e36d7e19ed"
format_stamp: "Formatted at 2026-09-27 22:05:29 on dist-test-slave-sg4j"
server_key: "2bfb6747c4227d71e8d47648206b8f43"
server_key_iv: "5d1a74d4f07630f8c1182ec841b38c3b"
server_key_version: "encryptionkey@0"
I20260927 22:06:16.770783 20560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:16.797515 20560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:16.803369 20560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:16.803719 20560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:16.805081 20560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:16.807096 20560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:16.807241 20560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:16.807518 20560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:16.807664 20560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:16.851341 20560 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:40879
I20260927 22:06:16.851471 20687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:40879 every 8 connection(s)
I20260927 22:06:16.853130 20560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:06:16.858237 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 20560
I20260927 22:06:16.859236 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:36771
--local_ip_for_outbound_sockets=127.3.159.3
--tserver_master_addrs=127.3.159.62:38479
--webserver_port=46455
--webserver_interface=127.3.159.3
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:42693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:06:16.860630 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:16.857228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:48361)
0927 22:06:16.858588 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:16.858971 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":533,"threads_started":1}
W20260927 22:06:16.862001 20688 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:38479 (0 consecutive failures): Network error: Failed to ping master at 127.3.159.62:38479: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
I20260927 22:06:16.863819 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:16.863202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:47617)
0927 22:06:16.863480 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:16.863611 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 22:06:16.865272 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:16.864731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:45897)
0927 22:06:16.864945 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:16.865064 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 22:06:16.865716 20688 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 22:06:17.282374 20691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:17.283070 20691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:17.283252 20691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 22:06:17.293145 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:17.291978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:55815)
0927 22:06:17.292685 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:17.292829 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":339,"threads_started":1}
W20260927 22:06:17.293438 20691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:17.293653 20691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:17.293854 20691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:06:17.306540 20691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:36771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=46455
--tserver_master_addrs=127.3.159.62:38479
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:17.307997 20691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:17.309829 20691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:17.325968 20697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:17.326344 20698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:17.326361 20700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:17.327628 20691 server_base.cc:1034] running on GCE node
I20260927 22:06:17.328558 20691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:17.329993 20691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:17.331284 20691 hybrid_clock.cc:648] HybridClock initialized: now 1790546777331222 us; error 74 us; skew 500 ppm
I20260927 22:06:17.331719 20691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:17 dist-test-slave-sg4j krb5kdc[19235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:17.335749 20691 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:06:17.337756 20691 webserver.cc:460] Webserver started at http://127.3.159.3:46455/ using document root <none> and password file <none>
I20260927 22:06:17.338553 20691 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:17.338812 20691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:17.345867 20691 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:06:17.349705 20707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:17.352092 20691 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:06:17.352387 20691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "32926f9156864f04a58b93c4b6b9078a"
format_stamp: "Formatted at 2026-09-27 22:05:30 on dist-test-slave-sg4j"
server_key: "e6402182a75ee53b28fd349d3ab3ae1d"
server_key_iv: "8705159ac96266c07818456c3d6dd096"
server_key_version: "encryptionkey@0"
I20260927 22:06:17.353665 20691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:17.372908 20691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:17.377988 20691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:17.378312 20691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:17.379536 20691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:17.381112 20691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:17.381213 20691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:17.381384 20691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:17.381515 20691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 22:06:17.424077 20691 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:36771
I20260927 22:06:17.424188 20819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:36771 every 8 connection(s)
I20260927 22:06:17.425800 20691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:06:17.432202 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:17.429787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:34571)
0927 22:06:17.430849 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:17.431132 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":408,"threads_started":1}
W20260927 22:06:17.433089 20820 heartbeater.cc:644] Failed to heartbeat to 127.3.159.62:38479 (0 consecutive failures): Network error: Failed to ping master at 127.3.159.62:38479: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
I20260927 22:06:17.434355 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:17.433824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:39651)
0927 22:06:17.434076 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:17.434194 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 22:06:17.435297 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 20691
I20260927 22:06:17.436019 20822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:17.435470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:34255)
0927 22:06:17.435688 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:17.435825 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 22:06:17.436331 20820 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 22:06:17.868096 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:17.866850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:57117)
0927 22:06:17.867574 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:17.867772 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":331,"threads_started":1}
I20260927 22:06:18.295645 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:18.294501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:58731)
0927 22:06:18.295243 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:18.295376 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":367,"threads_started":1}
I20260927 22:06:18.438627 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:18.437284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:58153)
0927 22:06:18.438116 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:18.438291 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":432,"threads_started":1}
I20260927 22:06:18.870796 20827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:18.869570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:33107)
0927 22:06:18.870443 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:18.870562 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":404,"threads_started":1}
I20260927 22:06:19.298245 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:19.297164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:48171)
0927 22:06:19.297906 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:19.298025 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":417,"threads_started":1}
I20260927 22:06:19.440864 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:19.439884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:50315)
0927 22:06:19.440544 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:19.440659 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":338,"threads_started":1}
I20260927 22:06:19.873277 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:19.872042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:50703)
0927 22:06:19.872717 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:19.872935 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":323,"threads_started":1}
I20260927 22:06:20.300587 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:20.299450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:38847)
0927 22:06:20.300231 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:20.300359 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":431,"threads_started":1}
I20260927 22:06:20.443578 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:20.442421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:58977)
0927 22:06:20.443157 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:20.443305 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":393,"threads_started":1}
I20260927 22:06:20.875754 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:20.874597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:37267)
0927 22:06:20.875316 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:20.875437 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":330,"threads_started":1}
I20260927 22:06:21.302963 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:21.301979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:34025)
0927 22:06:21.302617 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:21.302726 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":315,"threads_started":1}
I20260927 22:06:21.446074 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:21.445067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:53101)
0927 22:06:21.445759 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:21.445879 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":395,"threads_started":1}
I20260927 22:06:21.878314 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:21.877077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:54507)
0927 22:06:21.877923 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:21.878068 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":410,"threads_started":1}
I20260927 22:06:22.305150 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:22.304141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:48277)
0927 22:06:22.304851 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:22.304968 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":387,"threads_started":1}
I20260927 22:06:22.448349 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:22.447262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:52215)
0927 22:06:22.447989 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:22.448127 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":351,"threads_started":1}
I20260927 22:06:22.881143 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:22.880005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:56495)
0927 22:06:22.880774 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:22.880915 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":332,"threads_started":1}
I20260927 22:06:23.307827 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:23.306687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:45125)
0927 22:06:23.307361 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:23.307474 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":347,"threads_started":1}
I20260927 22:06:23.450686 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:23.449710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:56389)
0927 22:06:23.450362 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:23.450484 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":336,"threads_started":1}
I20260927 22:06:23.883795 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:23.882650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:34905)
0927 22:06:23.883390 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:23.883510 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":353,"threads_started":1}
I20260927 22:06:24.311053 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:24.309690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:49501)
0927 22:06:24.310434 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:24.310780 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":364,"threads_started":1}
I20260927 22:06:24.453599 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:24.452181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:47875)
0927 22:06:24.453078 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:24.453230 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":499,"threads_started":1}
I20260927 22:06:24.886392 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:24.885233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:59853)
0927 22:06:24.885989 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:24.886128 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":410,"threads_started":1}
I20260927 22:06:25.313202 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:25.312203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:46241)
0927 22:06:25.312898 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:25.313010 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":364,"threads_started":1}
I20260927 22:06:25.456063 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:25.455036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:33039)
0927 22:06:25.455725 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:25.455841 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":363,"threads_started":1}
I20260927 22:06:25.888690 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:25.887586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:57597)
0927 22:06:25.888323 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:25.888455 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":331,"threads_started":1}
I20260927 22:06:26.315688 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:26.314574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:49281)
0927 22:06:26.315395 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:26.315507 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":555,"threads_started":1}
I20260927 22:06:26.459118 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:26.457834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:53247)
0927 22:06:26.458691 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:26.458831 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":403,"threads_started":1}
I20260927 22:06:26.890973 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:26.890024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:49335)
0927 22:06:26.890663 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:26.890767 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":311,"threads_started":1}
I20260927 22:06:27.318396 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:27.317089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:37481)
0927 22:06:27.318018 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:27.318166 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":506,"threads_started":1}
I20260927 22:06:27.461346 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:27.460374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:60319)
0927 22:06:27.461019 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:27.461133 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":381,"threads_started":1}
I20260927 22:06:27.893630 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:27.892365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:38443)
0927 22:06:27.893128 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:27.893336 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":404,"threads_started":1}
I20260927 22:06:28.320958 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:28.320047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:53657)
0927 22:06:28.320654 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:28.320752 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":328,"threads_started":1}
I20260927 22:06:28.463721 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:28.462691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:34933)
0927 22:06:28.463363 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:28.463495 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":341,"threads_started":1}
I20260927 22:06:28.896054 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:28.894936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:42577)
0927 22:06:28.895661 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:28.895820 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":323,"threads_started":1}
I20260927 22:06:29.323859 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:29.322458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:37025)
0927 22:06:29.323217 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:29.323465 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":409,"threads_started":1}
I20260927 22:06:29.466071 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:29.465108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:56283)
0927 22:06:29.465792 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:29.465892 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":398,"threads_started":1}
I20260927 22:06:29.898484 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:29.897479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:49717)
0927 22:06:29.898133 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:29.898253 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":322,"threads_started":1}
I20260927 22:06:30.326243 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:30.325171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:49677)
0927 22:06:30.325908 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:30.326023 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":405,"threads_started":1}
I20260927 22:06:30.468924 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:30.467754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:43647)
0927 22:06:30.468484 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:30.468640 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":351,"threads_started":1}
I20260927 22:06:30.900765 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:30.899854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:54889)
0927 22:06:30.900488 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:30.900581 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":349,"threads_started":1}
I20260927 22:06:31.328876 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:31.327863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:48919)
0927 22:06:31.328478 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:31.328638 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":302,"threads_started":1}
I20260927 22:06:31.471873 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:31.470814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:49947)
0927 22:06:31.471506 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:31.471638 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":411,"threads_started":1}
I20260927 22:06:31.903525 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:31.902507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.2:46201)
0927 22:06:31.903159 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:31.903307 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":331,"threads_started":1}
I20260927 22:06:32.331393 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:32.330391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.1:50091)
0927 22:06:32.331072 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:32.331193 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":329,"threads_started":1}
I20260927 22:06:32.444561 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 20389
I20260927 22:06:32.466595 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:32.446717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.3.159.1:46743)
0927 22:06:32.447466 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:32.447481 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:32.447528 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:06:32.449285 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:32.449298 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:32.449860 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:32.450270 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:32.451982 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:32.453262 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:32.454689 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:32.455429 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:32.455648 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:32.459399 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:32.459415 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:32.459434 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:32.459489 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:32.462833 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:32.463659 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:32.463666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:32.463674 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:32.463873 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:32.464280 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:32.464285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:32.464291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:32.464913 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:32.465247 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:32.465747 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 22:06:32.466247 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":366,"threads_started":1}
I20260927 22:06:32.468719 20552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:32.476137 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:32.473741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:38479 (local address 127.3.159.3:36015)
0927 22:06:32.475694 (+ 1953us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:32.475869 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.62:38479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1790,"thread_start_us":1599,"threads_started":1}
I20260927 22:06:32.572521 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 20389
I20260927 22:06:32.592204 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 20560
I20260927 22:06:32.615744 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:32.593361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.3.159.2:40879)
0927 22:06:32.594252 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:32.594273 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:32.594325 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:06:32.596384 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:32.596397 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:32.597034 (+ 637us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:32.597562 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:32.599439 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:32.600985 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:32.602702 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:32.603612 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:32.603868 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:32.607409 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:32.607420 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:32.607446 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:32.607502 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:32.611262 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:32.612179 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:32.612187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:32.612196 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:32.612389 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:32.612918 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:32.612923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:32.612931 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:32.613771 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:32.614146 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:32.614846 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 22:06:32.615392 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":387,"threads_started":1}
I20260927 22:06:32.617908 20683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:32.707612 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 20560
I20260927 22:06:32.727032 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 20691
I20260927 22:06:32.750957 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:32.728313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39170 (local address 127.3.159.3:36771)
0927 22:06:32.729109 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:32.729129 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:32.729194 (+ 65us) server_negotiation.cc:407] Connection header received
0927 22:06:32.731326 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:32.731340 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:32.731946 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:32.732351 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:32.734261 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:32.735740 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:32.737470 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:32.738459 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:32.738675 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:32.741929 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:32.741941 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:32.741965 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:32.742058 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:32.745967 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:32.747202 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:32.747213 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:32.747223 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:32.747505 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:32.748121 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:32.748125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:32.748127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:32.748883 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:32.749216 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:32.749938 (+ 722us) server_negotiation.cc:299] Negotiation successful
0927 22:06:32.750561 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":372,"threads_started":1}
I20260927 22:06:32.752962 20815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:32.851370 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 20691
2026-09-27T22:06:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66879 ms)
[----------] 30 tests from SecurityITest (345561 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:06:32 dist-test-slave-sg4j krb5kdc[20877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:06:32 dist-test-slave-sg4j krb5kdc[20877](info): set up 2 sockets
Sep 27 22:06:32 dist-test-slave-sg4j krb5kdc[20877](info): commencing operation
krb5kdc: starting...
W20260927 22:06:34.038316 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:06:34 dist-test-slave-sg4j krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:06:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:06:34.375603 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:45323
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:45323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:06:34.818032 20890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:34.818539 20890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:34.818723 20890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:34.828361 20890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:06:34.828722 20890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:34.828860 20890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:34.828985 20890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:06:34.829074 20890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:06:34.842603 20890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:45323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:45323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:34.844681 20890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:34.847007 20890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:34.866222 20896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:34.866291 20898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:34.866291 20895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:34.868037 20890 server_base.cc:1034] running on GCE node
I20260927 22:06:34.869342 20890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:34.871567 20890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:34.873032 20890 hybrid_clock.cc:648] HybridClock initialized: now 1790546794872937 us; error 99 us; skew 500 ppm
I20260927 22:06:34.873660 20890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:34 dist-test-slave-sg4j krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546794, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:34.882323 20890 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:06:34.884725 20890 webserver.cc:460] Webserver started at http://127.3.159.62:35123/ using document root <none> and password file <none>
I20260927 22:06:34.885746 20890 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:34.885895 20890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:34.886198 20890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:34.891353 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "4718389002da4dfa965dc476e957962f"
format_stamp: "Formatted at 2026-09-27 22:06:34 on dist-test-slave-sg4j"
server_key: "966a45b578017700c7a29b3e3e52e7fe"
server_key_iv: "7c8294dfcf17fff795102b435f325164"
server_key_version: "encryptionkey@0"
I20260927 22:06:34.892508 20890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "4718389002da4dfa965dc476e957962f"
format_stamp: "Formatted at 2026-09-27 22:06:34 on dist-test-slave-sg4j"
server_key: "966a45b578017700c7a29b3e3e52e7fe"
server_key_iv: "7c8294dfcf17fff795102b435f325164"
server_key_version: "encryptionkey@0"
I20260927 22:06:34.902122 20890 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.009s
I20260927 22:06:34.910151 20905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:34.912981 20890 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260927 22:06:34.913285 20890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "4718389002da4dfa965dc476e957962f"
format_stamp: "Formatted at 2026-09-27 22:06:34 on dist-test-slave-sg4j"
server_key: "966a45b578017700c7a29b3e3e52e7fe"
server_key_iv: "7c8294dfcf17fff795102b435f325164"
server_key_version: "encryptionkey@0"
I20260927 22:06:34.913635 20890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:34.943576 20890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:34.949697 20890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:34.950019 20890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:34.982028 20890 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:45323
I20260927 22:06:34.982112 20956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:45323 every 8 connection(s)
I20260927 22:06:34.983990 20890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:06:34.989718 20957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:34.993723 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 20890
I20260927 22:06:34.994019 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:06:34.994989 3708 external_mini_cluster.cc:1428] Setting key bc406f9f522b5d2aed88b1141478cdd4
I20260927 22:06:35.008941 20957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f: Bootstrap starting.
I20260927 22:06:35.015318 20957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:35.017318 20957 log.cc:824] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:06:35 dist-test-slave-sg4j krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:35.023651 20957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f: No bootstrap required, opened a new log
I20260927 22:06:35.033162 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:34.998294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.3.159.62:45323)
0927 22:06:34.999404 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:34.999419 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:34.999478 (+ 59us) server_negotiation.cc:407] Connection header received
0927 22:06:35.001819 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:35.001857 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:35.002427 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:35.002936 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:35.005656 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:35.008345 (+ 2689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:35.011428 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:35.012477 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:35.012715 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:35.020087 (+ 7372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:35.020109 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:35.020141 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:35.020224 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:35.026567 (+ 6343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:35.028031 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:35.028046 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:35.028054 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:35.028367 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:35.028931 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:35.028936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:35.028938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:35.029883 (+ 945us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:35.030384 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:35.031186 (+ 802us) server_negotiation.cc:299] Negotiation successful
0927 22:06:35.031808 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":534,"threads_started":1}
I20260927 22:06:35.036955 20957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4718389002da4dfa965dc476e957962f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45323 } }
I20260927 22:06:35.037719 20957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:35.037887 20957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4718389002da4dfa965dc476e957962f, State: Initialized, Role: FOLLOWER
I20260927 22:06:35.038795 20957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4718389002da4dfa965dc476e957962f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45323 } }
I20260927 22:06:35.039136 20957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:06:35.039312 20957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:06:35.039558 20957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:35.042698 20957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4718389002da4dfa965dc476e957962f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45323 } }
I20260927 22:06:35.043560 20957 leader_election.cc:302] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4718389002da4dfa965dc476e957962f; no voters:
I20260927 22:06:35.044371 20957 leader_election.cc:288] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:06:35.044680 20962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:06:35.045872 20962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [term 1 LEADER]: Becoming Leader. State: Replica: 4718389002da4dfa965dc476e957962f, State: Running, Role: LEADER
I20260927 22:06:35.046703 20962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4718389002da4dfa965dc476e957962f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45323 } }
I20260927 22:06:35.047290 20957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:06:35.050432 20963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4718389002da4dfa965dc476e957962f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4718389002da4dfa965dc476e957962f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45323 } } }
I20260927 22:06:35.051740 20963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:35.051801 20964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4718389002da4dfa965dc476e957962f. Latest consensus state: current_term: 1 leader_uuid: "4718389002da4dfa965dc476e957962f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4718389002da4dfa965dc476e957962f" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45323 } } }
I20260927 22:06:35.052512 20964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:35.062652 20971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:06:35.072165 20971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:06:35.086460 20971 catalog_manager.cc:1347] Generated new cluster ID: 0b9ae27ba60541aab83781e69c9d3601
I20260927 22:06:35.086639 20971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:06:35.135542 20971 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:06:35.138950 20971 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:06:35.154021 20971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f: Generated new TSK 0
I20260927 22:06:35.155381 20971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:06:35.248742 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:45323
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:06:35.701465 20983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:35.701872 20983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:35.702069 20983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:35.711376 20983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:35.711588 20983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:35.712096 20983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:06:35.723610 20983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:45323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:35.725121 20983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:35.726609 20983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:35.745193 20988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:35.745263 20991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:35.745357 20989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:35.746536 20983 server_base.cc:1034] running on GCE node
I20260927 22:06:35.747396 20983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:35.748920 20983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:35.750263 20983 hybrid_clock.cc:648] HybridClock initialized: now 1790546795750187 us; error 61 us; skew 500 ppm
I20260927 22:06:35.750854 20983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:35 dist-test-slave-sg4j krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:35.755245 20983 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:06:35.757360 20983 webserver.cc:460] Webserver started at http://127.3.159.1:45625/ using document root <none> and password file <none>
I20260927 22:06:35.758641 20983 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:35.758841 20983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:35.759217 20983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:35.764094 20983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "c69eb2fc959f428ea75bff6ec0acbfd5"
format_stamp: "Formatted at 2026-09-27 22:06:35 on dist-test-slave-sg4j"
server_key: "859068106ec52ad54f2882c87920cee2"
server_key_iv: "b058dae92040de96113fd8ee073906d3"
server_key_version: "encryptionkey@0"
I20260927 22:06:35.765136 20983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "c69eb2fc959f428ea75bff6ec0acbfd5"
format_stamp: "Formatted at 2026-09-27 22:06:35 on dist-test-slave-sg4j"
server_key: "859068106ec52ad54f2882c87920cee2"
server_key_iv: "b058dae92040de96113fd8ee073906d3"
server_key_version: "encryptionkey@0"
I20260927 22:06:35.772462 20983 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260927 22:06:35.778702 20998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:35.780907 20983 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:06:35.781188 20983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "c69eb2fc959f428ea75bff6ec0acbfd5"
format_stamp: "Formatted at 2026-09-27 22:06:35 on dist-test-slave-sg4j"
server_key: "859068106ec52ad54f2882c87920cee2"
server_key_iv: "b058dae92040de96113fd8ee073906d3"
server_key_version: "encryptionkey@0"
I20260927 22:06:35.781589 20983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:35.799954 20983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:35.806422 20983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:35.806942 20983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:35.808162 20983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:35.810238 20983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:35.810370 20983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:35.810542 20983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:35.810622 20983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:35.856115 20983 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:42123
I20260927 22:06:35.856253 21110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:42123 every 8 connection(s)
I20260927 22:06:35.857658 20983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:06:35.860699 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 20983
I20260927 22:06:35.860965 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:06:35.861989 3708 external_mini_cluster.cc:1428] Setting key afba423a44ef00ff6502a8e2530ae4c8
Sep 27 22:06:35 dist-test-slave-sg4j krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546795, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:35.890173 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:35.862206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:54177 (local address 127.3.159.62:45323)
0927 22:06:35.862892 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:35.862898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:35.864629 (+ 1731us) server_negotiation.cc:407] Connection header received
0927 22:06:35.867583 (+ 2954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:35.867590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:35.867725 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:35.867871 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:35.869922 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:35.870890 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:35.872818 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:35.873244 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:35.873450 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:35.879398 (+ 5948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:35.879410 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:35.879419 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:35.879497 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:35.884105 (+ 4608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:35.885419 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:35.885429 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:35.885436 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:35.885706 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:35.886681 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:35.886691 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:35.886697 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:35.887271 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:35.887492 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:35.888960 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0927 22:06:35.889601 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":388,"threads_started":1}
I20260927 22:06:35.891260 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:35.863127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:45323 (local address 127.3.159.1:54177)
0927 22:06:35.864327 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:35.864442 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:35.867101 (+ 2659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:35.868187 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:35.868197 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:35.868818 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:35.869695 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:35.869719 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:35.871150 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:35.871154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:35.872595 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:35.872613 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:35.873577 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:35.873586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:35.873869 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:35.875299 (+ 1430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:06:35.875330 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:35.878839 (+ 3509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:35.884461 (+ 5622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:35.884472 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:35.884489 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:35.885128 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:35.885990 (+ 862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:35.886000 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:35.886006 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:35.886453 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:35.887596 (+ 1143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:35.887605 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:35.888357 (+ 752us) client_negotiation.cc:769] Sending connection context
0927 22:06:35.888741 (+ 384us) client_negotiation.cc:240] Negotiation successful
0927 22:06:35.889560 (+ 819us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":439,"threads_started":1}
I20260927 22:06:35.893599 21111 heartbeater.cc:347] Connected to a master server at 127.3.159.62:45323
I20260927 22:06:35.894084 21111 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:35.895133 21111 heartbeater.cc:511] Master 127.3.159.62:45323 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:06:35.898222 20922 ts_manager.cc:194] Registered new tserver with Master: c69eb2fc959f428ea75bff6ec0acbfd5 (127.3.159.1:42123)
I20260927 22:06:35.900019 20922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:54177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:06:35.943322 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:45323
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:06:36.337108 21117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:36.337536 21117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:36.337734 21117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:36.345716 21117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:36.345897 21117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:36.346094 21117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:06:36.357712 21117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:45323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:36.359444 21117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:36.361071 21117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:36.378022 21122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:36.378206 21125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:36.378124 21123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:36.379848 21117 server_base.cc:1034] running on GCE node
I20260927 22:06:36.380686 21117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:36.381979 21117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:36.383425 21117 hybrid_clock.cc:648] HybridClock initialized: now 1790546796383358 us; error 60 us; skew 500 ppm
I20260927 22:06:36.383874 21117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:36 dist-test-slave-sg4j krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:36.388360 21117 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:06:36.390906 21117 webserver.cc:460] Webserver started at http://127.3.159.2:39559/ using document root <none> and password file <none>
I20260927 22:06:36.391815 21117 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:36.392266 21117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:36.393353 21117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:36.396960 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "578d09469a9842cc8b728991a6be70cf"
format_stamp: "Formatted at 2026-09-27 22:06:36 on dist-test-slave-sg4j"
server_key: "c131f1c3a26fdb744f67f1fd74f9e470"
server_key_iv: "9260eea555b3683eb77d7dcaee31a122"
server_key_version: "encryptionkey@0"
I20260927 22:06:36.397980 21117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "578d09469a9842cc8b728991a6be70cf"
format_stamp: "Formatted at 2026-09-27 22:06:36 on dist-test-slave-sg4j"
server_key: "c131f1c3a26fdb744f67f1fd74f9e470"
server_key_iv: "9260eea555b3683eb77d7dcaee31a122"
server_key_version: "encryptionkey@0"
I20260927 22:06:36.404533 21117 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 22:06:36.410669 21132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:36.412917 21117 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:06:36.413142 21117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "578d09469a9842cc8b728991a6be70cf"
format_stamp: "Formatted at 2026-09-27 22:06:36 on dist-test-slave-sg4j"
server_key: "c131f1c3a26fdb744f67f1fd74f9e470"
server_key_iv: "9260eea555b3683eb77d7dcaee31a122"
server_key_version: "encryptionkey@0"
I20260927 22:06:36.413483 21117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:36.434247 21117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:36.441670 21117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:36.441985 21117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:36.443217 21117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:36.444890 21117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:36.444981 21117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:36.445072 21117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:36.445122 21117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:36.491326 21117 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:40807
I20260927 22:06:36.491377 21244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:40807 every 8 connection(s)
I20260927 22:06:36.493410 21117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:06:36.501803 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 21117
I20260927 22:06:36.502033 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:06:36.502605 3708 external_mini_cluster.cc:1428] Setting key eb1bdbe98845f15e654ddbd75ed3ce5a
Sep 27 22:06:36 dist-test-slave-sg4j krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:36.522192 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:36.497054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:44597 (local address 127.3.159.62:45323)
0927 22:06:36.497677 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:36.497682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:36.499042 (+ 1360us) server_negotiation.cc:407] Connection header received
0927 22:06:36.501142 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:36.501148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:36.501486 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:36.501667 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:36.504149 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:36.505238 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:36.507375 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:36.508271 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:36.508467 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:36.513322 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:36.513332 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:36.513341 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:36.513454 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:36.517317 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:36.518792 (+ 1475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:36.518797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:36.518800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:36.519019 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:36.519750 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:36.519755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:36.519757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:36.520263 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:36.520494 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:36.521630 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0927 22:06:36.521867 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":354,"threads_started":1}
I20260927 22:06:36.523684 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:36.497768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:45323 (local address 127.3.159.2:44597)
0927 22:06:36.498818 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:36.498877 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:36.500802 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:36.502163 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:36.502183 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:36.502961 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:36.503873 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:36.503906 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:36.505516 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:36.505521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:36.507118 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:36.507139 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:36.508617 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:36.508623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:36.508903 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:36.509831 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:06:36.509864 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:36.512914 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:36.517698 (+ 4784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:36.517712 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:36.517737 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:36.518559 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:36.519172 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:36.519180 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:36.519186 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:36.519512 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:36.520440 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:36.520452 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:36.521043 (+ 591us) client_negotiation.cc:769] Sending connection context
0927 22:06:36.521440 (+ 397us) client_negotiation.cc:240] Negotiation successful
0927 22:06:36.522094 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":409,"threads_started":1}
I20260927 22:06:36.525655 21245 heartbeater.cc:347] Connected to a master server at 127.3.159.62:45323
I20260927 22:06:36.526121 21245 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:36.527136 21245 heartbeater.cc:511] Master 127.3.159.62:45323 requested a full tablet report, sending...
I20260927 22:06:36.529367 20922 ts_manager.cc:194] Registered new tserver with Master: 578d09469a9842cc8b728991a6be70cf (127.3.159.2:40807)
I20260927 22:06:36.531207 20922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:44597
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:06:36.575459 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:45323
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:06:36.906648 21111 heartbeater.cc:503] Master 127.3.159.62:45323 was elected leader, sending a full tablet report...
W20260927 22:06:36.960961 21251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:36.961354 21251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:36.961560 21251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:36.970563 21251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:36.970723 21251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:36.970860 21251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:06:36.983263 21251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:45323
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:36.985093 21251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:36.987089 21251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:37.004354 21257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:37.004227 21256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:37.004429 21259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:37.006759 21251 server_base.cc:1034] running on GCE node
I20260927 22:06:37.007716 21251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:37.009167 21251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:37.010486 21251 hybrid_clock.cc:648] HybridClock initialized: now 1790546797010378 us; error 85 us; skew 500 ppm
I20260927 22:06:37.011093 21251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:37 dist-test-slave-sg4j krb5kdc[20877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:37.015890 21251 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:06:37.018404 21251 webserver.cc:460] Webserver started at http://127.3.159.3:37303/ using document root <none> and password file <none>
I20260927 22:06:37.019534 21251 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:37.019789 21251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:37.020272 21251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:37.024484 21251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "5e8f63fab1294b1a9068cb75ebd603bc"
format_stamp: "Formatted at 2026-09-27 22:06:37 on dist-test-slave-sg4j"
server_key: "0e7584ce4e1a8a5016d6b8201e874f83"
server_key_iv: "1c1113cbfb0a09faf256a60f1fb6c159"
server_key_version: "encryptionkey@0"
I20260927 22:06:37.025779 21251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "5e8f63fab1294b1a9068cb75ebd603bc"
format_stamp: "Formatted at 2026-09-27 22:06:37 on dist-test-slave-sg4j"
server_key: "0e7584ce4e1a8a5016d6b8201e874f83"
server_key_iv: "1c1113cbfb0a09faf256a60f1fb6c159"
server_key_version: "encryptionkey@0"
I20260927 22:06:37.033193 21251 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 22:06:37.038272 21266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:37.040736 21251 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:06:37.041050 21251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "5e8f63fab1294b1a9068cb75ebd603bc"
format_stamp: "Formatted at 2026-09-27 22:06:37 on dist-test-slave-sg4j"
server_key: "0e7584ce4e1a8a5016d6b8201e874f83"
server_key_iv: "1c1113cbfb0a09faf256a60f1fb6c159"
server_key_version: "encryptionkey@0"
I20260927 22:06:37.041353 21251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:37.087626 21251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:37.093226 21251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:37.093719 21251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:37.094990 21251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:37.096787 21251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:37.096913 21251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:37.097085 21251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:37.097164 21251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:37.140144 21251 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:44597
I20260927 22:06:37.140203 21378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:44597 every 8 connection(s)
I20260927 22:06:37.141847 21251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:06:37.150391 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 21251
I20260927 22:06:37.150637 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:06:37.151408 3708 external_mini_cluster.cc:1428] Setting key 245faee46430a07a3cfc920a34ad65a9
Sep 27 22:06:37 dist-test-slave-sg4j krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546797, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:37.170806 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.145412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:46255 (local address 127.3.159.62:45323)
0927 22:06:37.145998 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:37.146004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:37.147558 (+ 1554us) server_negotiation.cc:407] Connection header received
0927 22:06:37.150617 (+ 3059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:37.150625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:37.150781 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:37.150928 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:37.153718 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.154895 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.157335 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.157891 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.158066 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.162781 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:37.162789 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:37.162793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:37.162858 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:37.166020 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:37.167167 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:37.167173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:37.167176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:37.167412 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:37.168220 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:37.168232 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:37.168237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:37.168700 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:37.168919 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:37.170172 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0927 22:06:37.170433 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":279,"threads_started":1}
I20260927 22:06:37.171887 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.146056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:45323 (local address 127.3.159.3:46255)
0927 22:06:37.147266 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:37.147332 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:37.150007 (+ 2675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:37.151378 (+ 1371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:37.151404 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:37.152388 (+ 984us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:37.153439 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.153484 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.155395 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.155402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.157087 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.157114 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.158202 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.158210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.158518 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.159379 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:06:37.159409 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:37.162352 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:37.166287 (+ 3935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:37.166297 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:37.166317 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:37.166931 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:37.167624 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:37.167630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:37.167634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:37.168048 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:37.168996 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:37.169006 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:37.169712 (+ 706us) client_negotiation.cc:769] Sending connection context
0927 22:06:37.170040 (+ 328us) client_negotiation.cc:240] Negotiation successful
0927 22:06:37.170548 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":580,"threads_started":1}
I20260927 22:06:37.174103 21379 heartbeater.cc:347] Connected to a master server at 127.3.159.62:45323
I20260927 22:06:37.174547 21379 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:37.175680 21379 heartbeater.cc:511] Master 127.3.159.62:45323 requested a full tablet report, sending...
I20260927 22:06:37.177775 20922 ts_manager.cc:194] Registered new tserver with Master: 5e8f63fab1294b1a9068cb75ebd603bc (127.3.159.3:44597)
I20260927 22:06:37.178721 20922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:46255
I20260927 22:06:37.182771 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:06:37.204830 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.188501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.3.159.62:45323)
0927 22:06:37.188867 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:37.188876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:37.189302 (+ 426us) server_negotiation.cc:407] Connection header received
0927 22:06:37.189638 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:37.189643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:37.189758 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:37.189882 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:37.191568 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.192376 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.193959 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.194403 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.194557 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.197738 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:37.197744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:37.197748 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:37.197798 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:37.201145 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:37.202290 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:37.202296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:37.202299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:37.202503 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:37.202976 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:37.202983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:37.202991 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:37.203413 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:37.203742 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:37.204130 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 22:06:37.204451 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 22:06:37.211119 20922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:06:37.218713 20922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:06:37.266233 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.236683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.3.159.1:42123)
0927 22:06:37.249944 (+ 13261us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:37.249958 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:37.250785 (+ 827us) server_negotiation.cc:407] Connection header received
0927 22:06:37.251045 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:37.251052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:37.251274 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:37.251543 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:37.252611 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.255048 (+ 2437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.260130 (+ 5082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.262831 (+ 2701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.263122 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.263672 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:37.265608 (+ 1936us) server_negotiation.cc:299] Negotiation successful
0927 22:06:37.265838 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13039,"thread_start_us":13274,"threads_started":1}
I20260927 22:06:37.266757 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.236219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.3.159.2:40807)
0927 22:06:37.251627 (+ 15408us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:37.251641 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:37.251682 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:06:37.251816 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:37.251828 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:37.252170 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:37.252456 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:37.254184 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.256764 (+ 2580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.261348 (+ 4584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.263841 (+ 2493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.264079 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.264399 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:37.266125 (+ 1726us) server_negotiation.cc:299] Negotiation successful
0927 22:06:37.266401 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15175,"thread_start_us":15586,"threads_started":1}
I20260927 22:06:37.266763 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.235867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:40807 (local address 127.0.0.1:35006)
0927 22:06:37.250237 (+ 14370us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:37.250298 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:37.250549 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:37.252826 (+ 2277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:37.252832 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:37.252878 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:37.253803 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.253837 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.257133 (+ 3296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.257145 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.260914 (+ 3769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.260958 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.264668 (+ 3710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.264680 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.265570 (+ 890us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.265635 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 22:06:37.265770 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 22:06:37.266040 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13242,"mutex_wait_us":533,"spinlock_wait_cycles":17792,"thread_start_us":14005,"threads_started":1}
I20260927 22:06:37.267578 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.236455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:42123 (local address 127.0.0.1:59622)
0927 22:06:37.250237 (+ 13782us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:37.250298 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:37.250549 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:37.251544 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:37.251552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:37.251589 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:37.252148 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.252167 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.255397 (+ 3230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.255408 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.259694 (+ 4286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.259737 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.264005 (+ 4268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.264017 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.264424 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.264516 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 22:06:37.264770 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 22:06:37.267193 (+ 2423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13646,"mutex_wait_us":1703,"spinlock_wait_cycles":12928,"thread_start_us":12471,"threads_started":1}
I20260927 22:06:37.272158 21046 tablet_service.cc:1467] Processing CreateTablet for tablet 18a37e0f975a40d29292c0f4a75069d4 (DEFAULT_TABLE table=test-table [id=f9ab7d3dceda4b679f229dce84671cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:37.272606 21180 tablet_service.cc:1467] Processing CreateTablet for tablet 18a37e0f975a40d29292c0f4a75069d4 (DEFAULT_TABLE table=test-table [id=f9ab7d3dceda4b679f229dce84671cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:37.273998 21046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a37e0f975a40d29292c0f4a75069d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:37.274021 21180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a37e0f975a40d29292c0f4a75069d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:37.275713 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.252627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:44597 (local address 127.0.0.1:59752)
0927 22:06:37.267219 (+ 14592us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:37.267251 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:37.267452 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:37.269099 (+ 1647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:37.269104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:37.269138 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:37.269426 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.269439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.271549 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.271555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.272854 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.272867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.274783 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.274790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.274990 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.275026 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:06:37.275162 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 22:06:37.275340 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14413,"thread_start_us":12610,"threads_started":1}
I20260927 22:06:37.276257 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.252075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.3.159.3:44597)
0927 22:06:37.268238 (+ 16163us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:37.268249 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:37.268295 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:06:37.268429 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:37.268438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:37.268799 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:37.269627 (+ 828us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:37.269949 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.271380 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.273050 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.274599 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.274768 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.275079 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:37.275651 (+ 572us) server_negotiation.cc:299] Negotiation successful
0927 22:06:37.275904 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15954,"thread_start_us":15736,"threads_started":1}
I20260927 22:06:37.281471 21314 tablet_service.cc:1467] Processing CreateTablet for tablet 18a37e0f975a40d29292c0f4a75069d4 (DEFAULT_TABLE table=test-table [id=f9ab7d3dceda4b679f229dce84671cb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:37.283347 21314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a37e0f975a40d29292c0f4a75069d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:37.294554 21399 tablet_bootstrap.cc:492] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: Bootstrap starting.
I20260927 22:06:37.296316 21398 tablet_bootstrap.cc:492] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: Bootstrap starting.
I20260927 22:06:37.299587 21399 tablet_bootstrap.cc:654] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:37.300386 21398 tablet_bootstrap.cc:654] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:37.301602 21399 log.cc:824] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:37.302037 21398 log.cc:824] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:37.302282 21400 tablet_bootstrap.cc:492] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: Bootstrap starting.
I20260927 22:06:37.306942 21398 tablet_bootstrap.cc:492] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: No bootstrap required, opened a new log
I20260927 22:06:37.307104 21400 tablet_bootstrap.cc:654] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:37.307389 21398 ts_tablet_manager.cc:1397] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260927 22:06:37.307415 21399 tablet_bootstrap.cc:492] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: No bootstrap required, opened a new log
I20260927 22:06:37.307770 21399 ts_tablet_manager.cc:1397] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260927 22:06:37.309515 21400 log.cc:824] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:37.314695 21400 tablet_bootstrap.cc:492] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: No bootstrap required, opened a new log
I20260927 22:06:37.315084 21400 ts_tablet_manager.cc:1397] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260927 22:06:37.317734 21398 raft_consensus.cc:348] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578d09469a9842cc8b728991a6be70cf" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40807 } } peers { permanent_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44597 } } peers { permanent_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 42123 } }
I20260927 22:06:37.318277 21398 raft_consensus.cc:374] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:37.318380 21398 raft_consensus.cc:729] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 578d09469a9842cc8b728991a6be70cf, State: Initialized, Role: FOLLOWER
I20260927 22:06:37.319203 21398 consensus_queue.cc:260] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578d09469a9842cc8b728991a6be70cf" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40807 } } peers { permanent_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44597 } } peers { permanent_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 42123 } }
I20260927 22:06:37.319939 21399 raft_consensus.cc:348] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578d09469a9842cc8b728991a6be70cf" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40807 } } peers { permanent_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44597 } } peers { permanent_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 42123 } }
I20260927 22:06:37.320513 21399 raft_consensus.cc:374] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:37.320704 21399 raft_consensus.cc:729] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c69eb2fc959f428ea75bff6ec0acbfd5, State: Initialized, Role: FOLLOWER
I20260927 22:06:37.321537 21399 consensus_queue.cc:260] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578d09469a9842cc8b728991a6be70cf" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40807 } } peers { permanent_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44597 } } peers { permanent_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 42123 } }
I20260927 22:06:37.322001 21245 heartbeater.cc:503] Master 127.3.159.62:45323 was elected leader, sending a full tablet report...
I20260927 22:06:37.322954 21398 ts_tablet_manager.cc:1428] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260927 22:06:37.325243 21399 ts_tablet_manager.cc:1428] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: Time spent starting tablet: real 0.017s user 0.014s sys 0.004s
I20260927 22:06:37.328518 21400 raft_consensus.cc:348] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578d09469a9842cc8b728991a6be70cf" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40807 } } peers { permanent_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44597 } } peers { permanent_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 42123 } }
I20260927 22:06:37.329195 21400 raft_consensus.cc:374] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:37.329313 21400 raft_consensus.cc:729] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8f63fab1294b1a9068cb75ebd603bc, State: Initialized, Role: FOLLOWER
I20260927 22:06:37.330166 21400 consensus_queue.cc:260] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578d09469a9842cc8b728991a6be70cf" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40807 } } peers { permanent_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44597 } } peers { permanent_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 42123 } }
I20260927 22:06:37.332093 21379 heartbeater.cc:503] Master 127.3.159.62:45323 was elected leader, sending a full tablet report...
I20260927 22:06:37.332939 21400 ts_tablet_manager.cc:1428] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
W20260927 22:06:37.361896 21112 tablet.cc:2369] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:06:37.394160 21380 tablet.cc:2369] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:06:37.486929 21406 raft_consensus.cc:482] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:06:37.487402 21406 raft_consensus.cc:504] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578d09469a9842cc8b728991a6be70cf" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40807 } } peers { permanent_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44597 } } peers { permanent_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 42123 } }
I20260927 22:06:37.489933 21406 leader_election.cc:288] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 578d09469a9842cc8b728991a6be70cf (127.3.159.2:40807), c69eb2fc959f428ea75bff6ec0acbfd5 (127.3.159.1:42123)
W20260927 22:06:37.496449 21246 tablet.cc:2369] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:06:37.499542 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.490476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:40807 (local address 127.3.159.3:34235)
0927 22:06:37.490919 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:37.490944 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:37.491146 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:37.491838 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:37.491843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:37.491867 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:37.492110 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.492120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.493752 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.493760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.496228 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.496249 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.498500 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.498510 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.498775 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.498823 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 22:06:37.498978 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 22:06:37.499164 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260927 22:06:37.500026 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.491201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:35899 (local address 127.3.159.1:42123)
0927 22:06:37.491452 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:37.491462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:37.491837 (+ 375us) server_negotiation.cc:407] Connection header received
0927 22:06:37.492007 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:37.492014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:37.492101 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:37.492230 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:37.493037 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.494164 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.497276 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.498815 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.499013 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.499143 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:37.499546 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 22:06:37.499731 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:06:37.500118 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.490524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:34235 (local address 127.3.159.2:40807)
0927 22:06:37.490942 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:37.490950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:37.491098 (+ 148us) server_negotiation.cc:407] Connection header received
0927 22:06:37.491401 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:37.491408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:37.491510 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:37.491609 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:37.492273 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.493598 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.496499 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.498200 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.498380 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.498536 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:37.499260 (+ 724us) server_negotiation.cc:299] Negotiation successful
0927 22:06:37.499453 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260927 22:06:37.500898 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.490584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:42123 (local address 127.3.159.3:35899)
0927 22:06:37.491688 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:37.491716 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:37.491853 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:37.492326 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:37.492332 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:37.492367 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:37.492833 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.492845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.494374 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.494379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.497021 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:37.497038 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.499096 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.499102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:37.499316 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.499355 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 22:06:37.500300 (+ 945us) client_negotiation.cc:240] Negotiation successful
0927 22:06:37.500493 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"thread_start_us":849,"threads_started":1}
I20260927 22:06:37.501569 21200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a37e0f975a40d29292c0f4a75069d4" candidate_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578d09469a9842cc8b728991a6be70cf" is_pre_election: true
I20260927 22:06:37.502200 21200 raft_consensus.cc:2454] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8f63fab1294b1a9068cb75ebd603bc in term 0.
I20260927 22:06:37.502351 21066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a37e0f975a40d29292c0f4a75069d4" candidate_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" is_pre_election: true
I20260927 22:06:37.503127 21066 raft_consensus.cc:2454] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e8f63fab1294b1a9068cb75ebd603bc in term 0.
I20260927 22:06:37.503264 21268 leader_election.cc:302] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 578d09469a9842cc8b728991a6be70cf, 5e8f63fab1294b1a9068cb75ebd603bc; no voters:
I20260927 22:06:37.503860 21406 raft_consensus.cc:2781] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:06:37.504051 21406 raft_consensus.cc:482] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:06:37.504184 21406 raft_consensus.cc:3037] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:37.507087 21406 raft_consensus.cc:504] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578d09469a9842cc8b728991a6be70cf" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40807 } } peers { permanent_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44597 } } peers { permanent_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 42123 } }
I20260927 22:06:37.507990 21406 leader_election.cc:288] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [CANDIDATE]: Term 1 election: Requested vote from peers 578d09469a9842cc8b728991a6be70cf (127.3.159.2:40807), c69eb2fc959f428ea75bff6ec0acbfd5 (127.3.159.1:42123)
I20260927 22:06:37.508914 21200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a37e0f975a40d29292c0f4a75069d4" candidate_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578d09469a9842cc8b728991a6be70cf"
I20260927 22:06:37.509145 21066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a37e0f975a40d29292c0f4a75069d4" candidate_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5"
I20260927 22:06:37.509191 21200 raft_consensus.cc:3037] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:37.509743 21066 raft_consensus.cc:3037] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:37.512250 21200 raft_consensus.cc:2454] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8f63fab1294b1a9068cb75ebd603bc in term 1.
I20260927 22:06:37.512737 21066 raft_consensus.cc:2454] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e8f63fab1294b1a9068cb75ebd603bc in term 1.
I20260927 22:06:37.513031 21268 leader_election.cc:302] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 578d09469a9842cc8b728991a6be70cf, 5e8f63fab1294b1a9068cb75ebd603bc; no voters:
I20260927 22:06:37.513604 21406 raft_consensus.cc:2781] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:06:37.514547 21406 raft_consensus.cc:686] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 1 LEADER]: Becoming Leader. State: Replica: 5e8f63fab1294b1a9068cb75ebd603bc, State: Running, Role: LEADER
I20260927 22:06:37.515383 21406 consensus_queue.cc:237] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578d09469a9842cc8b728991a6be70cf" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40807 } } peers { permanent_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44597 } } peers { permanent_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 42123 } }
I20260927 22:06:37.522683 20921 catalog_manager.cc:5495] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e8f63fab1294b1a9068cb75ebd603bc (127.3.159.3). New cstate: current_term: 1 leader_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578d09469a9842cc8b728991a6be70cf" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e8f63fab1294b1a9068cb75ebd603bc" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 42123 } health_report { overall_health: UNKNOWN } } }
I20260927 22:06:37.591746 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.584091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59764 (local address 127.3.159.3:44597)
0927 22:06:37.584358 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:37.584369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:37.584529 (+ 160us) server_negotiation.cc:407] Connection header received
0927 22:06:37.584664 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:37.584668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:37.584761 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:37.585041 (+ 280us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:06:37.585761 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.586718 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.588801 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.589338 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.589664 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.590369 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:06:37.590960 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:06:37.591092 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:37.591382 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 22:06:37.591472 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 22:06:37.603489 21200 raft_consensus.cc:1261] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf [term 1 FOLLOWER]: Refusing update from remote peer 5e8f63fab1294b1a9068cb75ebd603bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:06:37.603571 21066 raft_consensus.cc:1261] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5 [term 1 FOLLOWER]: Refusing update from remote peer 5e8f63fab1294b1a9068cb75ebd603bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:06:37.605206 21408 consensus_queue.cc:1035] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "578d09469a9842cc8b728991a6be70cf" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:06:37.605790 21406 consensus_queue.cc:1035] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69eb2fc959f428ea75bff6ec0acbfd5" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:06:37.624820 21413 mvcc.cc:205] Tried to move back new op lower bound from 7334079682967904256 to 7334079682629394432. Current Snapshot: MvccSnapshot[applied={T|T < 7334079682967904256}]
I20260927 22:06:37.639344 20922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:06:37.639700 20922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:06:37.644116 20922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4718389002da4dfa965dc476e957962f: Sending DeleteTablet for 3 replicas of tablet 18a37e0f975a40d29292c0f4a75069d4
I20260927 22:06:37.645653 21046 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a37e0f975a40d29292c0f4a75069d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:37 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:59622
I20260927 22:06:37.645653 21180 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a37e0f975a40d29292c0f4a75069d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:37 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:35006
I20260927 22:06:37.645984 21314 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a37e0f975a40d29292c0f4a75069d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:37 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:59752
I20260927 22:06:37.649791 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 20983
I20260927 22:06:37.666755 21422 tablet_replica.cc:330] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: stopping tablet replica
I20260927 22:06:37.667519 21422 raft_consensus.cc:2229] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:06:37.668547 21422 raft_consensus.cc:2258] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:37.669066 21420 tablet_replica.cc:330] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: stopping tablet replica
I20260927 22:06:37.670389 21420 raft_consensus.cc:2229] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:06:37.672467 21420 raft_consensus.cc:2258] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:37.674098 21421 tablet_replica.cc:330] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: stopping tablet replica
I20260927 22:06:37.674985 21421 raft_consensus.cc:2229] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:06:37.675702 21421 raft_consensus.cc:2258] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 22:06:37 dist-test-slave-sg4j krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:06:37.676929 21422 ts_tablet_manager.cc:1905] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:37.679080 21421 ts_tablet_manager.cc:1905] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:37.686568 21422 ts_tablet_manager.cc:1918] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:37.686839 21422 log.cc:1196] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/18a37e0f975a40d29292c0f4a75069d4
I20260927 22:06:37.687474 21420 ts_tablet_manager.cc:1905] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:37.687603 21422 ts_tablet_manager.cc:1939] T 18a37e0f975a40d29292c0f4a75069d4 P 5e8f63fab1294b1a9068cb75ebd603bc: Deleting consensus metadata
I20260927 22:06:37.690326 20907 catalog_manager.cc:4841] TS 5e8f63fab1294b1a9068cb75ebd603bc (127.3.159.3:44597): tablet 18a37e0f975a40d29292c0f4a75069d4 (table test-table [id=f9ab7d3dceda4b679f229dce84671cb1]) successfully deleted
I20260927 22:06:37.691598 21421 ts_tablet_manager.cc:1918] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:37.692049 21421 log.cc:1196] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/18a37e0f975a40d29292c0f4a75069d4
I20260927 22:06:37.692780 21421 ts_tablet_manager.cc:1939] T 18a37e0f975a40d29292c0f4a75069d4 P 578d09469a9842cc8b728991a6be70cf: Deleting consensus metadata
I20260927 22:06:37.694576 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.651924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59624 (local address 127.3.159.1:42123)
0927 22:06:37.652193 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:37.652202 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:37.659410 (+ 7208us) server_negotiation.cc:407] Connection header received
0927 22:06:37.659916 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:37.659924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:37.660054 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:37.660145 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:37.662597 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.663963 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.669152 (+ 5189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.670150 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.670337 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.679149 (+ 8812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:37.679177 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:37.679200 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:37.679274 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:37.684163 (+ 4889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:37.690494 (+ 6331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:37.690510 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:37.690514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:37.690830 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:37.691568 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:37.691574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:37.691577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:37.692337 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:37.692757 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:37.693658 (+ 901us) server_negotiation.cc:299] Negotiation successful
0927 22:06:37.694046 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 22:06:37.695380 20907 catalog_manager.cc:4841] TS 578d09469a9842cc8b728991a6be70cf (127.3.159.2:40807): tablet 18a37e0f975a40d29292c0f4a75069d4 (table test-table [id=f9ab7d3dceda4b679f229dce84671cb1]) successfully deleted
I20260927 22:06:37.696199 21420 ts_tablet_manager.cc:1918] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:37.696449 21420 log.cc:1196] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/18a37e0f975a40d29292c0f4a75069d4
I20260927 22:06:37.696604 21106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:37.792960 21420 ts_tablet_manager.cc:1939] T 18a37e0f975a40d29292c0f4a75069d4 P c69eb2fc959f428ea75bff6ec0acbfd5: Deleting consensus metadata
I20260927 22:06:37.794175 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 20983
W20260927 22:06:37.818821 20909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.159.1:42123 (error 108)
I20260927 22:06:37.819728 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 21117
I20260927 22:06:37.821943 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.820537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:42123 (local address 127.0.0.1:59636)
0927 22:06:37.821509 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:37.821667 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":659,"threads_started":1}
W20260927 22:06:37.822494 20909 catalog_manager.cc:4584] TS c69eb2fc959f428ea75bff6ec0acbfd5 (127.3.159.1:42123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18a37e0f975a40d29292c0f4a75069d4: Network error: Client connection negotiation failed: client connection to 127.3.159.1:42123: connect: Connection refused (error 111)
Sep 27 22:06:37 dist-test-slave-sg4j krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:06:37.845712 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.820986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.3.159.2:40807)
0927 22:06:37.821232 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:37.821239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:37.821683 (+ 444us) server_negotiation.cc:407] Connection header received
0927 22:06:37.821927 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:37.821937 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:37.822049 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:37.822181 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:37.823915 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.824629 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.826105 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.826531 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.826686 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.837327 (+ 10641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:37.837347 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:37.837425 (+ 78us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:37.837494 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:37.842281 (+ 4787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:37.843268 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:37.843276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:37.843279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:37.843447 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:37.843872 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:37.843881 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:37.843883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:37.844320 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:37.844661 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:37.845025 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 22:06:37.845282 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 22:06:37.846654 21240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:37.876806 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.876003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:42123 (local address 127.0.0.1:59650)
0927 22:06:37.876390 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:37.876605 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260927 22:06:37.879065 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.878381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:42123 (local address 127.0.0.1:59666)
0927 22:06:37.878625 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:37.878773 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 22:06:37.940510 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.939764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:42123 (local address 127.0.0.1:59672)
0927 22:06:37.940055 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:37.940269 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 22:06:37.942468 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.941940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:42123 (local address 127.0.0.1:59674)
0927 22:06:37.942146 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:37.942252 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:06:37.951360 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 21117
I20260927 22:06:37.975492 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 21251
Sep 27 22:06:37 dist-test-slave-sg4j krb5kdc[20877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:06:38.002756 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:37.976803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.3.159.3:44597)
0927 22:06:37.977083 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:37.977091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:37.977121 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:06:37.977240 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:37.977247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:37.977426 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:37.977572 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:37.979163 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.980732 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.983746 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:37.984493 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:37.984708 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:37.991637 (+ 6929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:37.991663 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:37.991690 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:37.991766 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:37.997012 (+ 5246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:37.998719 (+ 1707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:37.998732 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:37.998736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:37.998997 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:37.999856 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:37.999864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:37.999869 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:38.000629 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:38.001009 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:38.001787 (+ 778us) server_negotiation.cc:299] Negotiation successful
0927 22:06:38.002305 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 22:06:38.003671 21374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:38.023734 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:38.023180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:42123 (local address 127.0.0.1:59690)
0927 22:06:38.023415 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:38.023537 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 22:06:38.025225 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:38.024654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:42123 (local address 127.0.0.1:59698)
0927 22:06:38.024898 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:38.025008 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260927 22:06:38.169629 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 21251
I20260927 22:06:38.195371 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 20890
I20260927 22:06:38.197268 20952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:38.328521 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:38.198100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:42123 (local address 127.0.0.1:59706)
0927 22:06:38.328005 (+129905us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:38.328234 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:42123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260927 22:06:38.328790 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 20890
2026-09-27T22:06:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5481 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T22:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:06:38Z Disabled control of system clock
I20260927 22:06:38.424765 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:44365
--webserver_interface=127.3.159.62
--webserver_port=0
--builtin_ntp_servers=127.3.159.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:44365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:06:38.825493 21433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:38.825891 21433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:38.825984 21433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:38.837689 21433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:06:38.837939 21433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:38.837986 21433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:38.838054 21433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:06:38.838107 21433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:06:38.849538 21433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:44365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:44365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:38.851030 21433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:38.853137 21433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:38.872175 21441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:38.872104 21438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:38.872123 21439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:38.873642 21433 server_base.cc:1034] running on GCE node
I20260927 22:06:38.874768 21433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:38.877007 21433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:38.878814 21433 hybrid_clock.cc:648] HybridClock initialized: now 1790546798878658 us; error 102 us; skew 500 ppm
I20260927 22:06:38.879434 21433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:06:38.882846 21433 webserver.cc:460] Webserver started at http://127.3.159.62:35299/ using document root <none> and password file <none>
I20260927 22:06:38.883766 21433 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:38.883898 21433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:38.884213 21433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:38.887902 21433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "bef7628f10b64070b7f866f7b34698b6"
format_stamp: "Formatted at 2026-09-27 22:06:38 on dist-test-slave-sg4j"
server_key: "20c571247e236fcd5ed66c98a68d9dbc"
server_key_iv: "9b23a78bf8ba0438277baebe73cc9be6"
server_key_version: "encryptionkey@0"
I20260927 22:06:38.889084 21433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "bef7628f10b64070b7f866f7b34698b6"
format_stamp: "Formatted at 2026-09-27 22:06:38 on dist-test-slave-sg4j"
server_key: "20c571247e236fcd5ed66c98a68d9dbc"
server_key_iv: "9b23a78bf8ba0438277baebe73cc9be6"
server_key_version: "encryptionkey@0"
I20260927 22:06:38.897370 21433 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.001s
I20260927 22:06:38.902794 21447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:38.905539 21433 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:06:38.905862 21433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "bef7628f10b64070b7f866f7b34698b6"
format_stamp: "Formatted at 2026-09-27 22:06:38 on dist-test-slave-sg4j"
server_key: "20c571247e236fcd5ed66c98a68d9dbc"
server_key_iv: "9b23a78bf8ba0438277baebe73cc9be6"
server_key_version: "encryptionkey@0"
I20260927 22:06:38.906227 21433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:38.928560 21433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:38.929603 21433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:38.929898 21433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:38.957047 21433 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:44365
I20260927 22:06:38.957105 21498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:44365 every 8 connection(s)
I20260927 22:06:38.959410 21433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:06:38.964511 21499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:38.968274 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 21433
I20260927 22:06:38.968624 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:06:38.969190 3708 external_mini_cluster.cc:1428] Setting key 0aef5b0e540945e774fc46b28ca7b796
I20260927 22:06:38.982821 21499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6: Bootstrap starting.
I20260927 22:06:38.988772 21499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:38.990684 21499 log.cc:824] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:38.991904 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:38.971874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.3.159.62:44365)
0927 22:06:38.973290 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:38.973306 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:38.973448 (+ 142us) server_negotiation.cc:407] Connection header received
0927 22:06:38.975970 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:38.976017 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:38.977175 (+ 1158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:38.978035 (+ 860us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:38.978922 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:38.981209 (+ 2287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:38.983894 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:38.985085 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:38.985352 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:38.987698 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:38.987710 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:38.987736 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:38.988554 (+ 818us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:38.988704 (+ 150us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:38.988744 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:38.988954 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:38.989594 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 22:06:38.990245 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":489,"threads_started":1}
I20260927 22:06:38.994756 21499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6: No bootstrap required, opened a new log
I20260927 22:06:39.005983 21499 raft_consensus.cc:348] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef7628f10b64070b7f866f7b34698b6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44365 } }
I20260927 22:06:39.007050 21499 raft_consensus.cc:374] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:39.007205 21499 raft_consensus.cc:729] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef7628f10b64070b7f866f7b34698b6, State: Initialized, Role: FOLLOWER
I20260927 22:06:39.008008 21499 consensus_queue.cc:260] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef7628f10b64070b7f866f7b34698b6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44365 } }
I20260927 22:06:39.008423 21499 raft_consensus.cc:388] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:06:39.008625 21499 raft_consensus.cc:482] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:06:39.008885 21499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:39.012988 21499 raft_consensus.cc:504] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef7628f10b64070b7f866f7b34698b6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44365 } }
I20260927 22:06:39.013775 21499 leader_election.cc:302] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bef7628f10b64070b7f866f7b34698b6; no voters:
I20260927 22:06:39.014599 21499 leader_election.cc:288] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:06:39.015081 21504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:06:39.016044 21504 raft_consensus.cc:686] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [term 1 LEADER]: Becoming Leader. State: Replica: bef7628f10b64070b7f866f7b34698b6, State: Running, Role: LEADER
I20260927 22:06:39.016901 21504 consensus_queue.cc:237] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef7628f10b64070b7f866f7b34698b6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44365 } }
I20260927 22:06:39.017194 21499 sys_catalog.cc:564] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:06:39.021484 21506 sys_catalog.cc:455] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bef7628f10b64070b7f866f7b34698b6. Latest consensus state: current_term: 1 leader_uuid: "bef7628f10b64070b7f866f7b34698b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef7628f10b64070b7f866f7b34698b6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44365 } } }
I20260927 22:06:39.022138 21506 sys_catalog.cc:458] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:39.021628 21505 sys_catalog.cc:455] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bef7628f10b64070b7f866f7b34698b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef7628f10b64070b7f866f7b34698b6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44365 } } }
I20260927 22:06:39.023312 21505 sys_catalog.cc:458] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:39.026363 21513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:06:39.035156 21513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:06:39.049536 21513 catalog_manager.cc:1347] Generated new cluster ID: 81aad50a5f8641258fde50c140fad095
I20260927 22:06:39.049731 21513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:06:39.078081 21513 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:06:39.081246 21513 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:06:39.100797 21513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6: Generated new TSK 0
I20260927 22:06:39.102187 21513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:06:39.118979 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:44365
--builtin_ntp_servers=127.3.159.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:06:39.668208 21523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:39.668618 21523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:39.668772 21523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:39.677567 21523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:39.677737 21523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:39.677934 21523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:06:39.691093 21523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:44365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:39.693260 21523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:39.696180 21523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:39.716174 21531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:39.716149 21529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:39.716482 21528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:39.717556 21523 server_base.cc:1034] running on GCE node
I20260927 22:06:39.718735 21523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:39.720453 21523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:39.724102 21523 hybrid_clock.cc:648] HybridClock initialized: now 1790546799723986 us; error 69 us; skew 500 ppm
I20260927 22:06:39.724758 21523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:06:39.728899 21523 webserver.cc:460] Webserver started at http://127.3.159.1:41775/ using document root <none> and password file <none>
I20260927 22:06:39.730544 21523 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:39.731021 21523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:39.731567 21523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:39.735680 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "b0f76a740eb745c3a65b9a038c33279b"
format_stamp: "Formatted at 2026-09-27 22:06:39 on dist-test-slave-sg4j"
server_key: "95eaf7d76fa2d483df3ff65ac6402988"
server_key_iv: "84cfc8bdd0a367c35b0a9f055263f879"
server_key_version: "encryptionkey@0"
I20260927 22:06:39.737725 21523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "b0f76a740eb745c3a65b9a038c33279b"
format_stamp: "Formatted at 2026-09-27 22:06:39 on dist-test-slave-sg4j"
server_key: "95eaf7d76fa2d483df3ff65ac6402988"
server_key_iv: "84cfc8bdd0a367c35b0a9f055263f879"
server_key_version: "encryptionkey@0"
I20260927 22:06:39.747809 21523 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
I20260927 22:06:39.754093 21537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:39.756590 21523 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20260927 22:06:39.756934 21523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "b0f76a740eb745c3a65b9a038c33279b"
format_stamp: "Formatted at 2026-09-27 22:06:39 on dist-test-slave-sg4j"
server_key: "95eaf7d76fa2d483df3ff65ac6402988"
server_key_iv: "84cfc8bdd0a367c35b0a9f055263f879"
server_key_version: "encryptionkey@0"
I20260927 22:06:39.757270 21523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:39.793427 21523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:39.794610 21523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:39.794975 21523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:39.796720 21523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:39.800172 21523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:39.800405 21523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:39.800603 21523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:39.800720 21523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:39.862166 21523 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:46031
I20260927 22:06:39.862588 21649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:46031 every 8 connection(s)
I20260927 22:06:39.863906 21523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:06:39.866596 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 21523
I20260927 22:06:39.866849 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:06:39.867551 3708 external_mini_cluster.cc:1428] Setting key bfc0ddfd4588fea9f515dc70ec6a03a2
I20260927 22:06:39.872184 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:44365
--builtin_ntp_servers=127.3.159.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:06:39.893522 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:39.867928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:55409 (local address 127.3.159.62:44365)
0927 22:06:39.868605 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:39.868613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:39.871073 (+ 2460us) server_negotiation.cc:407] Connection header received
0927 22:06:39.874533 (+ 3460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:39.874540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:39.874745 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:39.874909 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:39.877626 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:39.878902 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:39.882603 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:39.883141 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:39.883386 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:39.892109 (+ 8723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:39.892118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:39.892122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:39.892152 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:39.892222 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:39.892240 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:39.892361 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:39.892960 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 22:06:39.893110 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":365,"threads_started":1}
I20260927 22:06:39.895635 21653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:39.868814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44365 (local address 127.3.159.1:55409)
0927 22:06:39.870805 (+ 1991us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:39.870889 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:39.874005 (+ 3116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:39.875317 (+ 1312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:39.875335 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:39.875702 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:39.877104 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:39.877146 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:39.879277 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:39.879288 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:39.882209 (+ 2921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:39.882246 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:39.883911 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:39.883919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:39.884278 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:39.891079 (+ 6801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:06:39.891087 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:39.891207 (+ 120us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:06:39.891408 (+ 201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:39.892431 (+ 1023us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:39.892465 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:39.892481 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 22:06:39.892796 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 22:06:39.893618 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"thread_start_us":779,"threads_started":1}
I20260927 22:06:39.898242 21650 heartbeater.cc:347] Connected to a master server at 127.3.159.62:44365
I20260927 22:06:39.898953 21650 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:39.900554 21650 heartbeater.cc:511] Master 127.3.159.62:44365 requested a full tablet report, sending...
I20260927 22:06:39.904274 21464 ts_manager.cc:194] Registered new tserver with Master: b0f76a740eb745c3a65b9a038c33279b (127.3.159.1:46031)
I20260927 22:06:39.906419 21464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.1:55409
W20260927 22:06:40.366343 21654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:40.367044 21654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:40.367296 21654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:40.377049 21654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:40.377300 21654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:40.377553 21654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:06:40.390749 21654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:44365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:40.392736 21654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:40.396370 21654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:40.417539 21659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:40.417500 21662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:40.417444 21660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:40.419072 21654 server_base.cc:1034] running on GCE node
I20260927 22:06:40.420068 21654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:40.421751 21654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:40.423159 21654 hybrid_clock.cc:648] HybridClock initialized: now 1790546800423096 us; error 65 us; skew 500 ppm
I20260927 22:06:40.423730 21654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:06:40.428212 21654 webserver.cc:460] Webserver started at http://127.3.159.2:44161/ using document root <none> and password file <none>
I20260927 22:06:40.429672 21654 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:40.429864 21654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:40.430339 21654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:40.435596 21654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "6e77c460a5d24f2d98ee1c9683f0b82c"
format_stamp: "Formatted at 2026-09-27 22:06:40 on dist-test-slave-sg4j"
server_key: "0ab00a109f7ae68d1c89ea0f591b776e"
server_key_iv: "e998a2da726891c2ba8d958dde25db9d"
server_key_version: "encryptionkey@0"
I20260927 22:06:40.436770 21654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "6e77c460a5d24f2d98ee1c9683f0b82c"
format_stamp: "Formatted at 2026-09-27 22:06:40 on dist-test-slave-sg4j"
server_key: "0ab00a109f7ae68d1c89ea0f591b776e"
server_key_iv: "e998a2da726891c2ba8d958dde25db9d"
server_key_version: "encryptionkey@0"
I20260927 22:06:40.444734 21654 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260927 22:06:40.451124 21668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:40.453878 21654 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.002s
I20260927 22:06:40.454205 21654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "6e77c460a5d24f2d98ee1c9683f0b82c"
format_stamp: "Formatted at 2026-09-27 22:06:40 on dist-test-slave-sg4j"
server_key: "0ab00a109f7ae68d1c89ea0f591b776e"
server_key_iv: "e998a2da726891c2ba8d958dde25db9d"
server_key_version: "encryptionkey@0"
I20260927 22:06:40.454546 21654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:40.486483 21654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:40.487490 21654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:40.487813 21654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:40.489091 21654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:40.491046 21654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:40.491186 21654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:40.491338 21654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:40.491428 21654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:40.536607 21654 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:45149
I20260927 22:06:40.536729 21780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:45149 every 8 connection(s)
I20260927 22:06:40.538342 21654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:06:40.547901 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 21654
I20260927 22:06:40.548173 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:06:40.548723 3708 external_mini_cluster.cc:1428] Setting key 209a203ab550cca736a3c02573315d44
I20260927 22:06:40.553457 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:44365
--builtin_ntp_servers=127.3.159.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:06:40.558943 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:40.542466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:56565 (local address 127.3.159.62:44365)
0927 22:06:40.543195 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:40.543201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:40.544958 (+ 1757us) server_negotiation.cc:407] Connection header received
0927 22:06:40.547406 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:40.547414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:40.547627 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:40.547824 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:40.549639 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:40.550744 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:40.554075 (+ 3331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:40.555045 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:40.555261 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:40.557433 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:40.557442 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:40.557450 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:40.557499 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:40.557581 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:40.557605 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:40.557731 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:40.558157 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 22:06:40.558336 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":381,"threads_started":1}
I20260927 22:06:40.559911 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:40.543300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44365 (local address 127.3.159.2:56565)
0927 22:06:40.544734 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:40.544803 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:40.546932 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:40.548294 (+ 1362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:40.548316 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:40.548471 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:40.549269 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:40.549294 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:40.551017 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:40.551024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:40.553546 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:40.553577 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:40.555498 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:40.555508 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:40.555890 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:40.556883 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:06:40.556885 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:40.556939 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:06:40.557059 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:40.557841 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:40.557852 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:40.557860 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:06:40.558074 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 22:06:40.558508 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":417,"threads_started":1}
I20260927 22:06:40.562515 21781 heartbeater.cc:347] Connected to a master server at 127.3.159.62:44365
I20260927 22:06:40.563117 21781 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:40.564661 21781 heartbeater.cc:511] Master 127.3.159.62:44365 requested a full tablet report, sending...
I20260927 22:06:40.567559 21464 ts_manager.cc:194] Registered new tserver with Master: 6e77c460a5d24f2d98ee1c9683f0b82c (127.3.159.2:45149)
I20260927 22:06:40.568583 21464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.2:56565
I20260927 22:06:40.910740 21650 heartbeater.cc:503] Master 127.3.159.62:44365 was elected leader, sending a full tablet report...
W20260927 22:06:41.018935 21785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:41.019352 21785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:41.019510 21785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:41.028414 21785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:41.028597 21785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:41.028816 21785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:06:41.041504 21785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:44365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:41.043417 21785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:41.045820 21785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:41.065198 21791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:41.065240 21790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:41.065202 21793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:41.065780 21785 server_base.cc:1034] running on GCE node
I20260927 22:06:41.066957 21785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:41.068426 21785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:41.069816 21785 hybrid_clock.cc:648] HybridClock initialized: now 1790546801069695 us; error 106 us; skew 500 ppm
I20260927 22:06:41.070351 21785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:06:41.074519 21785 webserver.cc:460] Webserver started at http://127.3.159.3:34197/ using document root <none> and password file <none>
I20260927 22:06:41.075619 21785 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:41.075799 21785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:41.076231 21785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:41.080199 21785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "e8bae38484d9454481af433537da4ff3"
format_stamp: "Formatted at 2026-09-27 22:06:41 on dist-test-slave-sg4j"
server_key: "7ecec241194987e29f2d3d6830b48c3a"
server_key_iv: "7e8b13db6305e6a8f79079edb29c64b1"
server_key_version: "encryptionkey@0"
I20260927 22:06:41.081480 21785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "e8bae38484d9454481af433537da4ff3"
format_stamp: "Formatted at 2026-09-27 22:06:41 on dist-test-slave-sg4j"
server_key: "7ecec241194987e29f2d3d6830b48c3a"
server_key_iv: "7e8b13db6305e6a8f79079edb29c64b1"
server_key_version: "encryptionkey@0"
I20260927 22:06:41.090853 21785 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260927 22:06:41.097122 21799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:41.099748 21785 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:06:41.100118 21785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "e8bae38484d9454481af433537da4ff3"
format_stamp: "Formatted at 2026-09-27 22:06:41 on dist-test-slave-sg4j"
server_key: "7ecec241194987e29f2d3d6830b48c3a"
server_key_iv: "7e8b13db6305e6a8f79079edb29c64b1"
server_key_version: "encryptionkey@0"
I20260927 22:06:41.100502 21785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:41.130424 21785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:41.131778 21785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:41.132175 21785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:41.133770 21785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:41.135989 21785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:41.136155 21785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:41.136319 21785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:41.136415 21785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:41.183624 21785 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:44447
I20260927 22:06:41.183812 21911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:44447 every 8 connection(s)
I20260927 22:06:41.185429 21785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:06:41.195688 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 21785
I20260927 22:06:41.195955 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:06:41.196911 3708 external_mini_cluster.cc:1428] Setting key 54e4e86b3363adc8b50717421a9ea610
I20260927 22:06:41.207327 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.190507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:58175 (local address 127.3.159.62:44365)
0927 22:06:41.191235 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:41.191241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:41.193032 (+ 1791us) server_negotiation.cc:407] Connection header received
0927 22:06:41.195601 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:41.195609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:41.195744 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:41.195911 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:41.198437 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.199553 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.202733 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.203250 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.203392 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.205705 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:41.205712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:41.205716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:41.205749 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:41.205827 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:41.205846 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:41.205952 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:41.206368 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 22:06:41.206633 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":445,"threads_started":1}
I20260927 22:06:41.208352 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.191252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44365 (local address 127.3.159.3:58175)
0927 22:06:41.192747 (+ 1495us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:41.192805 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:41.195090 (+ 2285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:41.196422 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:41.196443 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:41.196885 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:41.198142 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.198174 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.199795 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.199800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.202386 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.202415 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.203704 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.203713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.204032 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.205018 (+ 986us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:06:41.205021 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:41.205077 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:06:41.205223 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:41.206014 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:41.206025 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:41.206033 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:06:41.206276 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 22:06:41.206902 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":441,"threads_started":1}
I20260927 22:06:41.213937 21912 heartbeater.cc:347] Connected to a master server at 127.3.159.62:44365
I20260927 22:06:41.214586 21912 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:41.216064 21912 heartbeater.cc:511] Master 127.3.159.62:44365 requested a full tablet report, sending...
I20260927 22:06:41.219336 21464 ts_manager.cc:194] Registered new tserver with Master: e8bae38484d9454481af433537da4ff3 (127.3.159.3:44447)
I20260927 22:06:41.221107 21464 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.3:58175
I20260927 22:06:41.233304 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:06:41.253072 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.241663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.3.159.62:44365)
0927 22:06:41.241932 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:41.241939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:41.242644 (+ 705us) server_negotiation.cc:407] Connection header received
0927 22:06:41.242766 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:41.242772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:41.242885 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:41.243160 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:41.244447 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.245453 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.248256 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.248843 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.249047 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.251254 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:41.251262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:41.251266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:41.251305 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:41.251389 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:41.251409 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:41.251949 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:41.252588 (+ 639us) server_negotiation.cc:299] Negotiation successful
0927 22:06:41.252747 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 22:06:41.261518 21464 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:06:41.272504 21464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:06:41.315601 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.295272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:44447 (local address 127.0.0.1:34280)
0927 22:06:41.296485 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:41.296553 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:41.296880 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:41.301895 (+ 5015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:41.301905 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:41.301949 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:41.302606 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.302625 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.306247 (+ 3622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.306252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.310455 (+ 4203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.310479 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.313719 (+ 3240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.313730 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.314031 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.314125 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 22:06:41.314665 (+ 540us) client_negotiation.cc:240] Negotiation successful
0927 22:06:41.314960 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":242,"thread_start_us":453,"threads_started":1}
I20260927 22:06:41.317968 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.295672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.3.159.3:44447)
0927 22:06:41.297789 (+ 2117us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:41.297813 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:41.297861 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:06:41.301337 (+ 3476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:41.301347 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:41.301614 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:41.303039 (+ 1425us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:41.303482 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.305226 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.310985 (+ 5759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.312736 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.315208 (+ 2472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.315855 (+ 647us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:41.316945 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0927 22:06:41.317469 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1886,"thread_start_us":1702,"threads_started":1}
I20260927 22:06:41.318177 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.297538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:45149 (local address 127.0.0.1:41698)
0927 22:06:41.304867 (+ 7329us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:41.304887 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:41.305122 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:41.306213 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:41.306220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:41.306264 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:41.308625 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.308645 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.310746 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.310756 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.313075 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.313087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.316884 (+ 3797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.316891 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.317138 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.317181 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:06:41.317308 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 22:06:41.317551 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7172,"thread_start_us":6975,"threads_started":1}
I20260927 22:06:41.319310 21930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.298745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.3.159.2:45149)
0927 22:06:41.304191 (+ 5446us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:41.304209 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:41.305043 (+ 834us) server_negotiation.cc:407] Connection header received
0927 22:06:41.305347 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:41.305359 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:41.305813 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:41.306258 (+ 445us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:41.308897 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.310435 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.313536 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.315775 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.316171 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.316538 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:41.318121 (+ 1583us) server_negotiation.cc:299] Negotiation successful
0927 22:06:41.318529 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2273,"thread_start_us":2034,"threads_started":1}
I20260927 22:06:41.320362 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.295523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.3.159.1:46031)
0927 22:06:41.300643 (+ 5120us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:41.300658 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:41.300703 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:06:41.301002 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:41.301012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:41.301487 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:41.301796 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:41.303165 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.305156 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.313405 (+ 8249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.315775 (+ 2370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.316226 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.316585 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:41.319385 (+ 2800us) server_negotiation.cc:299] Negotiation successful
0927 22:06:41.319943 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4790,"thread_start_us":4623,"threads_started":1}
I20260927 22:06:41.321271 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.295824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46031 (local address 127.0.0.1:36506)
0927 22:06:41.298883 (+ 3059us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:41.299021 (+ 138us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:41.299487 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:41.302112 (+ 2625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:41.302117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:41.302140 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:41.302887 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.302906 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.305369 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.305413 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.312949 (+ 7536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.312983 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.317740 (+ 4757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.317758 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.318761 (+ 1003us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.318830 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 22:06:41.320334 (+ 1504us) client_negotiation.cc:240] Negotiation successful
0927 22:06:41.320546 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2884,"mutex_wait_us":2,"thread_start_us":2067,"threads_started":1}
I20260927 22:06:41.325754 21847 tablet_service.cc:1467] Processing CreateTablet for tablet 9cebe2d117324d4db5aa442cd3893152 (DEFAULT_TABLE table=test-table [id=d91568e735a841aa9ec90e5afcf98a48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:41.326045 21716 tablet_service.cc:1467] Processing CreateTablet for tablet 9cebe2d117324d4db5aa442cd3893152 (DEFAULT_TABLE table=test-table [id=d91568e735a841aa9ec90e5afcf98a48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:41.328473 21847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cebe2d117324d4db5aa442cd3893152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:41.328584 21716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cebe2d117324d4db5aa442cd3893152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:41.328390 21585 tablet_service.cc:1467] Processing CreateTablet for tablet 9cebe2d117324d4db5aa442cd3893152 (DEFAULT_TABLE table=test-table [id=d91568e735a841aa9ec90e5afcf98a48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:41.330813 21585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cebe2d117324d4db5aa442cd3893152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:41.352818 21931 tablet_bootstrap.cc:492] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b: Bootstrap starting.
I20260927 22:06:41.356762 21932 tablet_bootstrap.cc:492] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: Bootstrap starting.
I20260927 22:06:41.358222 21933 tablet_bootstrap.cc:492] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: Bootstrap starting.
I20260927 22:06:41.359946 21931 tablet_bootstrap.cc:654] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:41.362540 21931 log.cc:824] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:41.364444 21932 tablet_bootstrap.cc:654] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:41.366398 21933 tablet_bootstrap.cc:654] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:41.366519 21932 log.cc:824] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:41.368424 21933 log.cc:824] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:41.369220 21931 tablet_bootstrap.cc:492] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b: No bootstrap required, opened a new log
I20260927 22:06:41.369745 21931 ts_tablet_manager.cc:1397] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260927 22:06:41.372084 21933 tablet_bootstrap.cc:492] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: No bootstrap required, opened a new log
I20260927 22:06:41.372237 21932 tablet_bootstrap.cc:492] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: No bootstrap required, opened a new log
I20260927 22:06:41.372772 21932 ts_tablet_manager.cc:1397] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20260927 22:06:41.372866 21933 ts_tablet_manager.cc:1397] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260927 22:06:41.386400 21933 raft_consensus.cc:348] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f76a740eb745c3a65b9a038c33279b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46031 } } peers { permanent_uuid: "e8bae38484d9454481af433537da4ff3" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44447 } } peers { permanent_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 45149 } }
I20260927 22:06:41.386958 21933 raft_consensus.cc:374] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:41.387095 21933 raft_consensus.cc:729] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e77c460a5d24f2d98ee1c9683f0b82c, State: Initialized, Role: FOLLOWER
I20260927 22:06:41.386634 21931 raft_consensus.cc:348] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f76a740eb745c3a65b9a038c33279b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46031 } } peers { permanent_uuid: "e8bae38484d9454481af433537da4ff3" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44447 } } peers { permanent_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 45149 } }
I20260927 22:06:41.387686 21931 raft_consensus.cc:374] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:41.387815 21931 raft_consensus.cc:729] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0f76a740eb745c3a65b9a038c33279b, State: Initialized, Role: FOLLOWER
I20260927 22:06:41.388737 21931 consensus_queue.cc:260] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f76a740eb745c3a65b9a038c33279b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46031 } } peers { permanent_uuid: "e8bae38484d9454481af433537da4ff3" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44447 } } peers { permanent_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 45149 } }
I20260927 22:06:41.389532 21933 consensus_queue.cc:260] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f76a740eb745c3a65b9a038c33279b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46031 } } peers { permanent_uuid: "e8bae38484d9454481af433537da4ff3" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44447 } } peers { permanent_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 45149 } }
I20260927 22:06:41.389880 21932 raft_consensus.cc:348] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f76a740eb745c3a65b9a038c33279b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46031 } } peers { permanent_uuid: "e8bae38484d9454481af433537da4ff3" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44447 } } peers { permanent_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 45149 } }
I20260927 22:06:41.390542 21932 raft_consensus.cc:374] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:41.390659 21932 raft_consensus.cc:729] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8bae38484d9454481af433537da4ff3, State: Initialized, Role: FOLLOWER
I20260927 22:06:41.391693 21931 ts_tablet_manager.cc:1428] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260927 22:06:41.391755 21932 consensus_queue.cc:260] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f76a740eb745c3a65b9a038c33279b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46031 } } peers { permanent_uuid: "e8bae38484d9454481af433537da4ff3" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44447 } } peers { permanent_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 45149 } }
I20260927 22:06:41.394351 21781 heartbeater.cc:503] Master 127.3.159.62:44365 was elected leader, sending a full tablet report...
I20260927 22:06:41.397369 21933 ts_tablet_manager.cc:1428] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: Time spent starting tablet: real 0.024s user 0.020s sys 0.003s
I20260927 22:06:41.397825 21912 heartbeater.cc:503] Master 127.3.159.62:44365 was elected leader, sending a full tablet report...
I20260927 22:06:41.398960 21932 ts_tablet_manager.cc:1428] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: Time spent starting tablet: real 0.026s user 0.019s sys 0.009s
W20260927 22:06:41.438671 21913 tablet.cc:2369] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:06:41.541775 21782 tablet.cc:2369] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:06:41.617852 21651 tablet.cc:2369] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:06:41.651518 21938 raft_consensus.cc:482] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:06:41.652050 21938 raft_consensus.cc:504] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f76a740eb745c3a65b9a038c33279b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46031 } } peers { permanent_uuid: "e8bae38484d9454481af433537da4ff3" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44447 } } peers { permanent_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 45149 } }
I20260927 22:06:41.654398 21938 leader_election.cc:288] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0f76a740eb745c3a65b9a038c33279b (127.3.159.1:46031), e8bae38484d9454481af433537da4ff3 (127.3.159.3:44447)
I20260927 22:06:41.664708 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.655223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:50155 (local address 127.3.159.1:46031)
0927 22:06:41.655502 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:41.655514 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:41.657080 (+ 1566us) server_negotiation.cc:407] Connection header received
0927 22:06:41.657437 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:41.657446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:41.657552 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:41.657686 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:41.658637 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.659738 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.662618 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.663588 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.663759 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.663882 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:41.664259 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 22:06:41.664412 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 22:06:41.664705 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.655239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:33653 (local address 127.3.159.3:44447)
0927 22:06:41.655494 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:41.655500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:41.657141 (+ 1641us) server_negotiation.cc:407] Connection header received
0927 22:06:41.657462 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:41.657473 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:41.657572 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:41.657699 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:41.658499 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.659438 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.662577 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.663552 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.663736 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.663864 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:41.664189 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 22:06:41.664358 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 22:06:41.665346 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.655760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:44447 (local address 127.3.159.2:33653)
0927 22:06:41.656948 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:41.656971 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:41.657225 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:41.657719 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:41.657723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:41.657752 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:41.658278 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.658293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.659633 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.659640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.662257 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.662281 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.663811 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.663819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.664007 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.664036 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:06:41.664153 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 22:06:41.664376 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":235,"thread_start_us":698,"threads_started":1}
I20260927 22:06:41.665460 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.655722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46031 (local address 127.3.159.2:50155)
0927 22:06:41.656872 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:41.656914 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:41.657177 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:41.657717 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:41.657723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:41.657836 (+ 113us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:41.658453 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.658465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.659908 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.659918 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.662276 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:41.662296 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.663872 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.663879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:41.664065 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.664097 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:06:41.664220 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 22:06:41.664443 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":118,"spinlock_wait_cycles":64384,"thread_start_us":643,"threads_started":1}
I20260927 22:06:41.666728 21605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cebe2d117324d4db5aa442cd3893152" candidate_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f76a740eb745c3a65b9a038c33279b" is_pre_election: true
I20260927 22:06:41.666878 21867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cebe2d117324d4db5aa442cd3893152" candidate_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bae38484d9454481af433537da4ff3" is_pre_election: true
I20260927 22:06:41.667366 21605 raft_consensus.cc:2454] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e77c460a5d24f2d98ee1c9683f0b82c in term 0.
I20260927 22:06:41.667528 21867 raft_consensus.cc:2454] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e77c460a5d24f2d98ee1c9683f0b82c in term 0.
I20260927 22:06:41.668279 21672 leader_election.cc:302] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6e77c460a5d24f2d98ee1c9683f0b82c, b0f76a740eb745c3a65b9a038c33279b, e8bae38484d9454481af433537da4ff3; no voters:
I20260927 22:06:41.668820 21938 raft_consensus.cc:2781] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:06:41.669008 21938 raft_consensus.cc:482] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:06:41.669140 21938 raft_consensus.cc:3037] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:41.672380 21938 raft_consensus.cc:504] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f76a740eb745c3a65b9a038c33279b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46031 } } peers { permanent_uuid: "e8bae38484d9454481af433537da4ff3" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44447 } } peers { permanent_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 45149 } }
I20260927 22:06:41.673466 21938 leader_election.cc:288] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [CANDIDATE]: Term 1 election: Requested vote from peers b0f76a740eb745c3a65b9a038c33279b (127.3.159.1:46031), e8bae38484d9454481af433537da4ff3 (127.3.159.3:44447)
I20260927 22:06:41.674166 21867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cebe2d117324d4db5aa442cd3893152" candidate_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8bae38484d9454481af433537da4ff3"
I20260927 22:06:41.674155 21605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cebe2d117324d4db5aa442cd3893152" candidate_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0f76a740eb745c3a65b9a038c33279b"
I20260927 22:06:41.674430 21867 raft_consensus.cc:3037] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:41.674458 21605 raft_consensus.cc:3037] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:41.677209 21867 raft_consensus.cc:2454] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e77c460a5d24f2d98ee1c9683f0b82c in term 1.
I20260927 22:06:41.677239 21605 raft_consensus.cc:2454] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e77c460a5d24f2d98ee1c9683f0b82c in term 1.
I20260927 22:06:41.678339 21670 leader_election.cc:302] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6e77c460a5d24f2d98ee1c9683f0b82c, b0f76a740eb745c3a65b9a038c33279b, e8bae38484d9454481af433537da4ff3; no voters:
I20260927 22:06:41.678880 21938 raft_consensus.cc:2781] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:06:41.679553 21938 raft_consensus.cc:686] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 1 LEADER]: Becoming Leader. State: Replica: 6e77c460a5d24f2d98ee1c9683f0b82c, State: Running, Role: LEADER
I20260927 22:06:41.680160 21938 consensus_queue.cc:237] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f76a740eb745c3a65b9a038c33279b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46031 } } peers { permanent_uuid: "e8bae38484d9454481af433537da4ff3" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44447 } } peers { permanent_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 45149 } }
I20260927 22:06:41.686064 21462 catalog_manager.cc:5495] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e77c460a5d24f2d98ee1c9683f0b82c (127.3.159.2). New cstate: current_term: 1 leader_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0f76a740eb745c3a65b9a038c33279b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8bae38484d9454481af433537da4ff3" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e77c460a5d24f2d98ee1c9683f0b82c" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 45149 } health_report { overall_health: HEALTHY } } }
I20260927 22:06:41.793586 21930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.786091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.3.159.2:45149)
0927 22:06:41.786361 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:41.786366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:41.786800 (+ 434us) server_negotiation.cc:407] Connection header received
0927 22:06:41.787020 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:41.787025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:41.787142 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:41.787312 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:06:41.787890 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.788807 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.790685 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.791248 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.791438 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.792128 (+ 690us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:06:41.792811 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:06:41.793024 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:41.793080 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 22:06:41.793191 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:06:41.805074 21605 raft_consensus.cc:1261] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b [term 1 FOLLOWER]: Refusing update from remote peer 6e77c460a5d24f2d98ee1c9683f0b82c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:06:41.805291 21867 raft_consensus.cc:1261] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3 [term 1 FOLLOWER]: Refusing update from remote peer 6e77c460a5d24f2d98ee1c9683f0b82c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:06:41.806679 21942 consensus_queue.cc:1035] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8bae38484d9454481af433537da4ff3" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 44447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:06:41.807204 21938 consensus_queue.cc:1035] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0f76a740eb745c3a65b9a038c33279b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 46031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:06:41.831816 21948 mvcc.cc:205] Tried to move back new op lower bound from 7334079700178231296 to 7334079699686985728. Current Snapshot: MvccSnapshot[applied={T|T < 7334079700178231296}]
I20260927 22:06:41.848636 21462 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:06:41.849014 21462 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:06:41.854424 21462 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bef7628f10b64070b7f866f7b34698b6: Sending DeleteTablet for 3 replicas of tablet 9cebe2d117324d4db5aa442cd3893152
I20260927 22:06:41.856194 21847 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cebe2d117324d4db5aa442cd3893152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:41 UTC) from {username='slave'} at 127.0.0.1:34280
I20260927 22:06:41.856197 21585 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cebe2d117324d4db5aa442cd3893152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:41 UTC) from {username='slave'} at 127.0.0.1:36506
I20260927 22:06:41.856356 21716 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cebe2d117324d4db5aa442cd3893152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:41 UTC) from {username='slave'} at 127.0.0.1:41698
I20260927 22:06:41.858624 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 21523
I20260927 22:06:41.872437 21957 tablet_replica.cc:330] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: stopping tablet replica
I20260927 22:06:41.873104 21957 raft_consensus.cc:2229] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:06:41.874045 21957 raft_consensus.cc:2258] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:41.876477 21957 ts_tablet_manager.cc:1905] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:41.885131 21957 ts_tablet_manager.cc:1918] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:41.885567 21957 log.cc:1196] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/9cebe2d117324d4db5aa442cd3893152
I20260927 22:06:41.886296 21957 ts_tablet_manager.cc:1939] T 9cebe2d117324d4db5aa442cd3893152 P 6e77c460a5d24f2d98ee1c9683f0b82c: Deleting consensus metadata
I20260927 22:06:41.888542 21449 catalog_manager.cc:4841] TS 6e77c460a5d24f2d98ee1c9683f0b82c (127.3.159.2:45149): tablet 9cebe2d117324d4db5aa442cd3893152 (table test-table [id=d91568e735a841aa9ec90e5afcf98a48]) successfully deleted
I20260927 22:06:41.893076 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:41.861154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.3.159.1:46031)
0927 22:06:41.865679 (+ 4525us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:41.865692 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:41.871500 (+ 5808us) server_negotiation.cc:407] Connection header received
0927 22:06:41.871752 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:41.871762 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:41.871965 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:41.872174 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:41.874081 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.875729 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.885740 (+ 10011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:41.886801 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:41.887102 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:41.890082 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:41.890101 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:41.890121 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:41.891246 (+ 1125us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:41.891325 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:41.891360 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:41.891492 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:41.892219 (+ 727us) server_negotiation.cc:299] Negotiation successful
0927 22:06:41.892384 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4265}
I20260927 22:06:41.894418 21645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:41.902288 21955 tablet_replica.cc:330] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: stopping tablet replica
I20260927 22:06:41.903703 21955 raft_consensus.cc:2229] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:06:41.904528 21955 raft_consensus.cc:2258] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:41.907951 21955 ts_tablet_manager.cc:1905] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:41.919261 21955 ts_tablet_manager.cc:1918] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:41.919575 21955 log.cc:1196] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/9cebe2d117324d4db5aa442cd3893152
I20260927 22:06:41.920455 21955 ts_tablet_manager.cc:1939] T 9cebe2d117324d4db5aa442cd3893152 P e8bae38484d9454481af433537da4ff3: Deleting consensus metadata
I20260927 22:06:41.922883 21449 catalog_manager.cc:4841] TS e8bae38484d9454481af433537da4ff3 (127.3.159.3:44447): tablet 9cebe2d117324d4db5aa442cd3893152 (table test-table [id=d91568e735a841aa9ec90e5afcf98a48]) successfully deleted
I20260927 22:06:42.043607 21956 tablet_replica.cc:330] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b: stopping tablet replica
I20260927 22:06:42.044387 21956 raft_consensus.cc:2229] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:06:42.045092 21956 raft_consensus.cc:2258] T 9cebe2d117324d4db5aa442cd3893152 P b0f76a740eb745c3a65b9a038c33279b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:42.045995 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 21523
I20260927 22:06:42.068359 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 21654
W20260927 22:06:42.068444 21451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.159.1:46031 (error 108)
I20260927 22:06:42.071472 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:42.070347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46031 (local address 127.0.0.1:36526)
0927 22:06:42.070965 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:42.071170 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":304,"threads_started":1}
W20260927 22:06:42.071885 21451 catalog_manager.cc:4584] TS b0f76a740eb745c3a65b9a038c33279b (127.3.159.1:46031): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cebe2d117324d4db5aa442cd3893152: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46031: connect: Connection refused (error 111)
I20260927 22:06:42.078966 21930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:42.069950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.3.159.2:45149)
0927 22:06:42.070203 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:42.070210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:42.070238 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:06:42.070311 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:42.070316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:42.070442 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:42.070585 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:42.071400 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:42.072784 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:42.074685 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:42.075478 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:42.075679 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:42.077036 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:42.077048 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:42.077070 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:42.078018 (+ 948us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:42.078091 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:42.078120 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:42.078262 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:42.078429 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 22:06:42.078565 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 22:06:42.080003 21776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:42.126497 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:42.125853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46031 (local address 127.0.0.1:36534)
0927 22:06:42.126187 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:42.126300 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 22:06:42.127671 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:42.127240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46031 (local address 127.0.0.1:36536)
0927 22:06:42.127416 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:42.127503 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 22:06:42.188692 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:42.187921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46031 (local address 127.0.0.1:36538)
0927 22:06:42.188289 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:42.188458 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260927 22:06:42.190332 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:42.189742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46031 (local address 127.0.0.1:36540)
0927 22:06:42.189962 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:42.190094 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 22:06:42.208941 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 21654
I20260927 22:06:42.231909 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 21785
I20260927 22:06:42.241937 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:42.233006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.3.159.3:44447)
0927 22:06:42.233761 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:42.233767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:42.233786 (+ 19us) server_negotiation.cc:407] Connection header received
0927 22:06:42.233855 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:42.233859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:42.233963 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:42.234071 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:42.234667 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:42.235545 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:42.237129 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:42.237861 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:42.237977 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:42.239402 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:42.239417 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:42.239443 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:42.240641 (+ 1198us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:42.240736 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:42.240762 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:42.240869 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:42.241146 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 22:06:42.241320 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":398,"threads_started":1}
I20260927 22:06:42.243024 21907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:42.271844 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:42.271180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46031 (local address 127.0.0.1:36556)
0927 22:06:42.271515 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:42.271641 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260927 22:06:42.273649 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:42.272790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46031 (local address 127.0.0.1:36568)
0927 22:06:42.273094 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:42.273200 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260927 22:06:42.347448 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 21785
I20260927 22:06:42.366403 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 21433
I20260927 22:06:42.367920 21494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:42.489583 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:42.488875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:46031 (local address 127.0.0.1:36580)
0927 22:06:42.489162 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:42.489291 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:46031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260927 22:06:42.489651 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 21433
2026-09-27T22:06:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (4160 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:06:42 dist-test-slave-sg4j krb5kdc[21968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:06:42 dist-test-slave-sg4j krb5kdc[21968](info): set up 2 sockets
Sep 27 22:06:42 dist-test-slave-sg4j krb5kdc[21968](info): commencing operation
krb5kdc: starting...
W20260927 22:06:43.682166 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.142s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:06:43 dist-test-slave-sg4j krb5kdc[21968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:06:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:06:44.009984 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:36389
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:36389
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:06:44.472934 21981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:44.473371 21981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:44.473606 21981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:44.482347 21981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:06:44.482535 21981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:44.482590 21981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:44.482697 21981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:06:44.482781 21981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:06:44.494964 21981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:36389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:36389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:44.496814 21981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:44.498704 21981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:44.514705 21986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:44.514585 21987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:44.514716 21989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:44.516220 21981 server_base.cc:1034] running on GCE node
I20260927 22:06:44.517532 21981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:44.519717 21981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:44.521005 21981 hybrid_clock.cc:648] HybridClock initialized: now 1790546804520940 us; error 50 us; skew 500 ppm
I20260927 22:06:44.521582 21981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:44 dist-test-slave-sg4j krb5kdc[21968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546804, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:44.526078 21981 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:06:44.528280 21981 webserver.cc:460] Webserver started at http://127.3.159.62:42587/ using document root <none> and password file <none>
I20260927 22:06:44.529157 21981 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:44.529287 21981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:44.529707 21981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:44.533107 21981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "1fb770b202054162819b5643097b6734"
format_stamp: "Formatted at 2026-09-27 22:06:44 on dist-test-slave-sg4j"
server_key: "93920ce937998f715187bc8c223117bd"
server_key_iv: "a6dbe608b5a0dd008b95894f821ef49e"
server_key_version: "encryptionkey@0"
I20260927 22:06:44.534035 21981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "1fb770b202054162819b5643097b6734"
format_stamp: "Formatted at 2026-09-27 22:06:44 on dist-test-slave-sg4j"
server_key: "93920ce937998f715187bc8c223117bd"
server_key_iv: "a6dbe608b5a0dd008b95894f821ef49e"
server_key_version: "encryptionkey@0"
I20260927 22:06:44.541730 21981 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260927 22:06:44.546906 21996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:44.549103 21981 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:06:44.549453 21981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "1fb770b202054162819b5643097b6734"
format_stamp: "Formatted at 2026-09-27 22:06:44 on dist-test-slave-sg4j"
server_key: "93920ce937998f715187bc8c223117bd"
server_key_iv: "a6dbe608b5a0dd008b95894f821ef49e"
server_key_version: "encryptionkey@0"
I20260927 22:06:44.549723 21981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:44.584594 21981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:44.590130 21981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:44.590517 21981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:44.616544 21981 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:36389
I20260927 22:06:44.616607 22047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:36389 every 8 connection(s)
I20260927 22:06:44.618337 21981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:06:44.624022 22048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:44.626953 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 21981
I20260927 22:06:44.627231 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:06:44.627856 3708 external_mini_cluster.cc:1428] Setting key b9b826c31db3a55b7bad96a6081b3d97
I20260927 22:06:44.639762 22048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734: Bootstrap starting.
I20260927 22:06:44.644061 22048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734: Neither blocks nor log segments found. Creating new log.
Sep 27 22:06:44 dist-test-slave-sg4j krb5kdc[21968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:44.645568 22048 log.cc:824] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:44.650806 22048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734: No bootstrap required, opened a new log
I20260927 22:06:44.659253 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:44.630482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58116 (local address 127.3.159.62:36389)
0927 22:06:44.631646 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:44.631663 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:44.631720 (+ 57us) server_negotiation.cc:407] Connection header received
0927 22:06:44.634004 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:44.634041 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:44.634581 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:44.635093 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:44.636819 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:44.638424 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:44.640088 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:44.641118 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:44.641396 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:44.646938 (+ 5542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:44.646950 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:44.646973 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:44.647042 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:44.652258 (+ 5216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:44.653445 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:44.653457 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:44.653469 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:44.653757 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:44.654349 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:44.654355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:44.654359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:44.655459 (+ 1100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:44.655861 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:44.656545 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 22:06:44.657227 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":504,"threads_started":1}
I20260927 22:06:44.663271 22048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb770b202054162819b5643097b6734" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36389 } }
I20260927 22:06:44.663921 22048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:44.664067 22048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb770b202054162819b5643097b6734, State: Initialized, Role: FOLLOWER
I20260927 22:06:44.664852 22048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb770b202054162819b5643097b6734" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36389 } }
I20260927 22:06:44.665210 22048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:06:44.665434 22048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:06:44.665668 22048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:44.668545 22048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb770b202054162819b5643097b6734" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36389 } }
I20260927 22:06:44.669356 22048 leader_election.cc:302] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fb770b202054162819b5643097b6734; no voters:
I20260927 22:06:44.670428 22048 leader_election.cc:288] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:06:44.670862 22053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:06:44.672097 22053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [term 1 LEADER]: Becoming Leader. State: Replica: 1fb770b202054162819b5643097b6734, State: Running, Role: LEADER
I20260927 22:06:44.672854 22053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb770b202054162819b5643097b6734" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36389 } }
I20260927 22:06:44.673456 22048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:06:44.676218 22055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fb770b202054162819b5643097b6734. Latest consensus state: current_term: 1 leader_uuid: "1fb770b202054162819b5643097b6734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb770b202054162819b5643097b6734" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36389 } } }
I20260927 22:06:44.676707 22055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:44.677824 22054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fb770b202054162819b5643097b6734" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb770b202054162819b5643097b6734" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36389 } } }
I20260927 22:06:44.678193 22054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734 [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:44.678359 22058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:06:44.696630 22058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:06:44.711432 22058 catalog_manager.cc:1347] Generated new cluster ID: a26e6e814c7f4553960d90762457245a
I20260927 22:06:44.711619 22058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:06:44.728003 22058 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:06:44.729871 22058 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:06:44.746104 22058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734: Generated new TSK 0
I20260927 22:06:44.747448 22058 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:06:44.839282 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36389
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:06:45.284089 22074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:45.284412 22074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:45.284508 22074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:45.293015 22074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:45.293187 22074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:45.293880 22074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:06:45.307435 22074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:45.309662 22074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:45.311625 22074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:45.330247 22082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:45.330353 22080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:45.330726 22079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:45.332907 22074 server_base.cc:1034] running on GCE node
I20260927 22:06:45.334089 22074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:45.335364 22074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:45.336714 22074 hybrid_clock.cc:648] HybridClock initialized: now 1790546805336629 us; error 68 us; skew 500 ppm
I20260927 22:06:45.337328 22074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:45 dist-test-slave-sg4j krb5kdc[21968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546805, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:45.348443 22074 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:06:45.350649 22074 webserver.cc:460] Webserver started at http://127.3.159.1:37871/ using document root <none> and password file <none>
I20260927 22:06:45.351572 22074 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:45.351715 22074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:45.352097 22074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:45.356259 22074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "333eccd9f57146349ad9371c6cb17694"
format_stamp: "Formatted at 2026-09-27 22:06:45 on dist-test-slave-sg4j"
server_key: "bc69c4ab8ce16bfa2b636ab7597e7b8a"
server_key_iv: "9860b57e49037dc337f95b077fcf2265"
server_key_version: "encryptionkey@0"
I20260927 22:06:45.357518 22074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "333eccd9f57146349ad9371c6cb17694"
format_stamp: "Formatted at 2026-09-27 22:06:45 on dist-test-slave-sg4j"
server_key: "bc69c4ab8ce16bfa2b636ab7597e7b8a"
server_key_iv: "9860b57e49037dc337f95b077fcf2265"
server_key_version: "encryptionkey@0"
I20260927 22:06:45.364776 22074 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 22:06:45.370249 22089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:45.372941 22074 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:06:45.373255 22074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "333eccd9f57146349ad9371c6cb17694"
format_stamp: "Formatted at 2026-09-27 22:06:45 on dist-test-slave-sg4j"
server_key: "bc69c4ab8ce16bfa2b636ab7597e7b8a"
server_key_iv: "9860b57e49037dc337f95b077fcf2265"
server_key_version: "encryptionkey@0"
I20260927 22:06:45.373605 22074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:45.389063 22074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:45.396791 22074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:45.397181 22074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:45.398522 22074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:45.400303 22074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:45.400426 22074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:45.400599 22074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:45.400674 22074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:45.443997 22074 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:34787
I20260927 22:06:45.444320 22201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:34787 every 8 connection(s)
I20260927 22:06:45.445708 22074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:06:45.449208 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 22074
I20260927 22:06:45.449662 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:06:45.450585 3708 external_mini_cluster.cc:1428] Setting key 9643ee81a6cb41d00149409d735451a0
Sep 27 22:06:45 dist-test-slave-sg4j krb5kdc[21968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546805, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:45.478379 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:45.450307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:35427 (local address 127.3.159.62:36389)
0927 22:06:45.451193 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:45.451199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:45.452878 (+ 1679us) server_negotiation.cc:407] Connection header received
0927 22:06:45.456285 (+ 3407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:45.456295 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:45.456435 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:45.456685 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:45.459747 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:45.460980 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:45.463148 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:45.463932 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:45.464125 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:45.469491 (+ 5366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:45.469497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:45.469501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:45.469545 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:45.472967 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:45.474448 (+ 1481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:45.474459 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:45.474464 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:45.474763 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:45.475487 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:45.475493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:45.475495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:45.475904 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:45.476114 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:45.477262 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0927 22:06:45.477856 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":597,"threads_started":1}
I20260927 22:06:45.479811 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:45.451170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:36389 (local address 127.3.159.1:35427)
0927 22:06:45.452564 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:45.452658 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:45.455749 (+ 3091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:45.457109 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:45.457126 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:45.458273 (+ 1147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:45.459389 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:45.459420 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:45.461235 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:45.461241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:45.462920 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:45.462937 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:45.464249 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:45.464254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:45.464524 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:45.465583 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:06:45.465633 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:45.469096 (+ 3463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:45.473287 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:45.473301 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:45.473341 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:45.473996 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:45.474962 (+ 966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:45.474969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:45.474973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:45.475303 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:45.476073 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:45.476085 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:45.476648 (+ 563us) client_negotiation.cc:769] Sending connection context
0927 22:06:45.477009 (+ 361us) client_negotiation.cc:240] Negotiation successful
0927 22:06:45.478186 (+ 1177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"thread_start_us":524,"threads_started":1}
I20260927 22:06:45.482340 22202 heartbeater.cc:347] Connected to a master server at 127.3.159.62:36389
I20260927 22:06:45.482908 22202 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:45.484233 22202 heartbeater.cc:511] Master 127.3.159.62:36389 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:06:45.488325 22013 ts_manager.cc:194] Registered new tserver with Master: 333eccd9f57146349ad9371c6cb17694 (127.3.159.1:34787)
I20260927 22:06:45.490554 22013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:35427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:06:45.526278 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36389
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:06:45.959889 22208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:45.960323 22208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:45.960518 22208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:45.969662 22208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:45.969846 22208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:45.970062 22208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:06:45.983727 22208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:45.985781 22208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:45.987612 22208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:46.006738 22213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:46.007110 22214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:46.007110 22216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:46.008287 22208 server_base.cc:1034] running on GCE node
I20260927 22:06:46.009227 22208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:46.011153 22208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:46.012718 22208 hybrid_clock.cc:648] HybridClock initialized: now 1790546806012636 us; error 64 us; skew 500 ppm
I20260927 22:06:46.013252 22208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:46 dist-test-slave-sg4j krb5kdc[21968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:46.017805 22208 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:06:46.019873 22208 webserver.cc:460] Webserver started at http://127.3.159.2:35937/ using document root <none> and password file <none>
I20260927 22:06:46.021049 22208 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:46.021286 22208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:46.021731 22208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:46.025331 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "b092306be5fc4976b235b6a488c00a9d"
format_stamp: "Formatted at 2026-09-27 22:06:46 on dist-test-slave-sg4j"
server_key: "3e70e98a1c8b182d593b8d2cc07a88bd"
server_key_iv: "e292b42f17b427d863b6abab4e22ae7a"
server_key_version: "encryptionkey@0"
I20260927 22:06:46.026335 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "b092306be5fc4976b235b6a488c00a9d"
format_stamp: "Formatted at 2026-09-27 22:06:46 on dist-test-slave-sg4j"
server_key: "3e70e98a1c8b182d593b8d2cc07a88bd"
server_key_iv: "e292b42f17b427d863b6abab4e22ae7a"
server_key_version: "encryptionkey@0"
I20260927 22:06:46.033087 22208 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 22:06:46.038784 22223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:46.041050 22208 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:06:46.041337 22208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "b092306be5fc4976b235b6a488c00a9d"
format_stamp: "Formatted at 2026-09-27 22:06:46 on dist-test-slave-sg4j"
server_key: "3e70e98a1c8b182d593b8d2cc07a88bd"
server_key_iv: "e292b42f17b427d863b6abab4e22ae7a"
server_key_version: "encryptionkey@0"
I20260927 22:06:46.041678 22208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:46.062336 22208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:46.068939 22208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:46.069331 22208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:46.070669 22208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:46.072649 22208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:46.072849 22208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:46.073045 22208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:46.073235 22208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:46.122515 22208 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:37049
I20260927 22:06:46.122643 22335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:37049 every 8 connection(s)
I20260927 22:06:46.124153 22208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:06:46.132392 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 22208
I20260927 22:06:46.132656 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:06:46.133329 3708 external_mini_cluster.cc:1428] Setting key 145ac3a036a132077311a706ea50a297
Sep 27 22:06:46 dist-test-slave-sg4j krb5kdc[21968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:46.163393 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:46.128183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:39141 (local address 127.3.159.62:36389)
0927 22:06:46.128765 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:46.128770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:46.130877 (+ 2107us) server_negotiation.cc:407] Connection header received
0927 22:06:46.134518 (+ 3641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:46.134525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:46.134674 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:46.134837 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:46.137654 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.138466 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.142608 (+ 4142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.143302 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.143466 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:46.150406 (+ 6940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:46.150417 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:46.150423 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:46.150503 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:46.156610 (+ 6107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:46.158677 (+ 2067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:46.158688 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:46.158695 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:46.158977 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:46.159976 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:46.159981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:46.159984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:46.160653 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:46.160956 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:46.162316 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0927 22:06:46.162871 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":313,"threads_started":1}
I20260927 22:06:46.164968 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:46.128903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:36389 (local address 127.3.159.2:39141)
0927 22:06:46.130563 (+ 1660us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:46.130651 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:46.133926 (+ 3275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:46.135252 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:46.135267 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:46.136078 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:46.137262 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:46.137308 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.138804 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.138814 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:46.142145 (+ 3331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:46.142188 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.143707 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.143718 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:46.144172 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:46.145614 (+ 1442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:06:46.145674 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:46.149881 (+ 4207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:46.157028 (+ 7147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:46.157044 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:46.157074 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:46.158308 (+ 1234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:46.159290 (+ 982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:46.159296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:46.159300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:46.159668 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:46.160955 (+ 1287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:46.160969 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:46.161739 (+ 770us) client_negotiation.cc:769] Sending connection context
0927 22:06:46.162165 (+ 426us) client_negotiation.cc:240] Negotiation successful
0927 22:06:46.163294 (+ 1129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"thread_start_us":873,"threads_started":1}
I20260927 22:06:46.167093 22336 heartbeater.cc:347] Connected to a master server at 127.3.159.62:36389
I20260927 22:06:46.167886 22336 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:46.168828 22336 heartbeater.cc:511] Master 127.3.159.62:36389 requested a full tablet report, sending...
I20260927 22:06:46.170933 22013 ts_manager.cc:194] Registered new tserver with Master: b092306be5fc4976b235b6a488c00a9d (127.3.159.2:37049)
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
I20260927 22:06:46.171926 22013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:39141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:06:46.215415 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36389
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:06:46.495200 22202 heartbeater.cc:503] Master 127.3.159.62:36389 was elected leader, sending a full tablet report...
W20260927 22:06:46.686946 22342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:46.687364 22342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:46.687542 22342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:46.696439 22342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:46.696623 22342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:46.696842 22342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:06:46.709913 22342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:36389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:46.711853 22342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:46.714094 22342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:46.734433 22347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:46.734323 22350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:46.734404 22348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:46.735633 22342 server_base.cc:1034] running on GCE node
I20260927 22:06:46.736574 22342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:46.738199 22342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:46.739598 22342 hybrid_clock.cc:648] HybridClock initialized: now 1790546806739438 us; error 100 us; skew 500 ppm
I20260927 22:06:46.740156 22342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:46 dist-test-slave-sg4j krb5kdc[21968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:46.745803 22342 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:06:46.748651 22342 webserver.cc:460] Webserver started at http://127.3.159.3:36161/ using document root <none> and password file <none>
I20260927 22:06:46.749833 22342 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:46.750036 22342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:46.750450 22342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:46.754539 22342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "a576d10a5fef42139f4b09ed87b5529f"
format_stamp: "Formatted at 2026-09-27 22:06:46 on dist-test-slave-sg4j"
server_key: "944da7030b5b0336819cffeeb32a013b"
server_key_iv: "69beb6a886cee48122600ba638b0f7c5"
server_key_version: "encryptionkey@0"
I20260927 22:06:46.755601 22342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "a576d10a5fef42139f4b09ed87b5529f"
format_stamp: "Formatted at 2026-09-27 22:06:46 on dist-test-slave-sg4j"
server_key: "944da7030b5b0336819cffeeb32a013b"
server_key_iv: "69beb6a886cee48122600ba638b0f7c5"
server_key_version: "encryptionkey@0"
I20260927 22:06:46.762984 22342 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 22:06:46.768272 22357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:46.770629 22342 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 22:06:46.770876 22342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "a576d10a5fef42139f4b09ed87b5529f"
format_stamp: "Formatted at 2026-09-27 22:06:46 on dist-test-slave-sg4j"
server_key: "944da7030b5b0336819cffeeb32a013b"
server_key_iv: "69beb6a886cee48122600ba638b0f7c5"
server_key_version: "encryptionkey@0"
I20260927 22:06:46.771131 22342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:46.795967 22342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:46.802671 22342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:46.803031 22342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:46.804435 22342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:46.806448 22342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:46.806572 22342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:46.806694 22342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:46.806739 22342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:46.856328 22342 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:34383
I20260927 22:06:46.856504 22469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:34383 every 8 connection(s)
I20260927 22:06:46.858371 22342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:06:46.865947 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 22342
I20260927 22:06:46.866181 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:06:46.866850 3708 external_mini_cluster.cc:1428] Setting key be678d292171291cabb6d5c499002b11
Sep 27 22:06:46 dist-test-slave-sg4j krb5kdc[21968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546806, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:46.889994 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:46.862485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:43117 (local address 127.3.159.62:36389)
0927 22:06:46.863286 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:46.863295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:46.865088 (+ 1793us) server_negotiation.cc:407] Connection header received
0927 22:06:46.868260 (+ 3172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:46.868268 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:46.868472 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:46.868599 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:46.871363 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.872396 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.875532 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.876202 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.876399 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:46.882120 (+ 5721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:46.882129 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:46.882134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:46.882205 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:46.886144 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:46.887164 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:46.887169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:46.887171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:46.887309 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:46.887819 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:46.887823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:46.887825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:46.888134 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:46.888324 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:46.889117 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 22:06:46.889557 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":333,"threads_started":1}
I20260927 22:06:46.891287 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:46.863433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:36389 (local address 127.3.159.3:43117)
0927 22:06:46.864776 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:46.864861 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:46.867885 (+ 3024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:46.869082 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:46.869109 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:46.869951 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:46.871012 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:46.871050 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.872753 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.872760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:46.875192 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:46.875224 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.876586 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.876594 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:46.877010 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:46.878417 (+ 1407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:06:46.878457 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:46.881581 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:46.886404 (+ 4823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:46.886421 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:46.886448 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:46.886995 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:46.887437 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:46.887454 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:46.887459 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:46.887671 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:46.888290 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:46.888301 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:46.888716 (+ 415us) client_negotiation.cc:769] Sending connection context
0927 22:06:46.889019 (+ 303us) client_negotiation.cc:240] Negotiation successful
0927 22:06:46.889827 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":460,"threads_started":1}
I20260927 22:06:46.894055 22470 heartbeater.cc:347] Connected to a master server at 127.3.159.62:36389
I20260927 22:06:46.894760 22470 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:46.895962 22470 heartbeater.cc:511] Master 127.3.159.62:36389 requested a full tablet report, sending...
I20260927 22:06:46.898279 22013 ts_manager.cc:194] Registered new tserver with Master: a576d10a5fef42139f4b09ed87b5529f (127.3.159.3:34383)
I20260927 22:06:46.899498 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:06:46.899590 22013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:43117
I20260927 22:06:46.926003 22472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:46.907521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58112 (local address 127.3.159.62:36389)
0927 22:06:46.907851 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:46.907860 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:46.907906 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:06:46.908018 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:46.908026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:46.908143 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:46.908305 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:46.911425 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.913125 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.915352 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.915786 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.915951 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:46.919271 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:46.919279 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:46.919284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:46.919350 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:46.922591 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:46.923637 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:46.923643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:46.923646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:46.923796 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:46.924296 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:46.924300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:46.924305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:46.924662 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:46.924985 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:46.925460 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 22:06:46.925682 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 22:06:46.932629 22013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:06:46.941897 22013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:06:46.977798 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:46.962828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:34787 (local address 127.0.0.1:55478)
0927 22:06:46.965145 (+ 2317us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:46.965166 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:46.965303 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:46.967092 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:46.967101 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:46.967140 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:46.967961 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:46.967982 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.970641 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.970647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:46.973620 (+ 2973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:46.973648 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.976419 (+ 2771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.976427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:46.976695 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:46.976773 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 22:06:46.976914 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 22:06:46.977224 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2172,"thread_start_us":2014,"threads_started":1}
I20260927 22:06:46.979883 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:46.963532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.3.159.1:34787)
0927 22:06:46.966012 (+ 2480us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:46.966025 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:46.966188 (+ 163us) server_negotiation.cc:407] Connection header received
0927 22:06:46.966336 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:46.966347 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:46.966738 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:46.968283 (+ 1545us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:46.968610 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.970427 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.973969 (+ 3542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.976073 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.978675 (+ 2602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:46.979055 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:46.979170 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 22:06:46.979424 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2255,"thread_start_us":2771,"threads_started":1}
I20260927 22:06:46.982338 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:46.962073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:37049 (local address 127.0.0.1:35418)
0927 22:06:46.964231 (+ 2158us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:46.964285 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:46.964586 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:46.968555 (+ 3969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:46.968561 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:46.968592 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:46.969134 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:46.969150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.973830 (+ 4680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.973836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:46.978917 (+ 5081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:46.978934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.981322 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.981331 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:46.981598 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:46.981641 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 22:06:46.981740 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 22:06:46.981917 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1827,"thread_start_us":1641,"threads_started":1}
I20260927 22:06:46.983915 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:46.962472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.3.159.2:37049)
0927 22:06:46.963493 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:46.963506 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:46.964871 (+ 1365us) server_negotiation.cc:407] Connection header received
0927 22:06:46.967444 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:46.967557 (+ 113us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:46.967923 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:46.968235 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:46.969442 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.973538 (+ 4096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.979209 (+ 5671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.981019 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.981214 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:46.983018 (+ 1804us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:46.983150 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 22:06:46.983513 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809,"thread_start_us":624,"threads_started":1}
I20260927 22:06:46.985791 22137 tablet_service.cc:1467] Processing CreateTablet for tablet 519d76fb214542a4a9ad9c9cccfa35dc (DEFAULT_TABLE table=test-table [id=5a650c3db8f64bf8b29467271ef31028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:46.987659 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:46.965532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:34383 (local address 127.0.0.1:56306)
0927 22:06:46.978157 (+ 12625us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:46.978181 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:46.978315 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:46.980256 (+ 1941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:46.980263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:46.980302 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:46.980505 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:46.980515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.982759 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.982765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:46.984332 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:46.984348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.986778 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.986785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:46.986998 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:46.987038 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:06:46.987176 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 22:06:46.987347 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12459,"thread_start_us":13282,"threads_started":1}
I20260927 22:06:46.987886 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:46.965975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56306 (local address 127.3.159.3:34383)
0927 22:06:46.979114 (+ 13139us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:46.979127 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:46.979174 (+ 47us) server_negotiation.cc:407] Connection header received
0927 22:06:46.979308 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:46.979321 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:46.979851 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:46.980669 (+ 818us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:46.980924 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.982545 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.984796 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:46.986516 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:46.986708 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:46.986997 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:46.987255 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 22:06:46.987476 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12490,"thread_start_us":12716,"threads_started":1}
I20260927 22:06:46.988375 22137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519d76fb214542a4a9ad9c9cccfa35dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:46.988686 22271 tablet_service.cc:1467] Processing CreateTablet for tablet 519d76fb214542a4a9ad9c9cccfa35dc (DEFAULT_TABLE table=test-table [id=5a650c3db8f64bf8b29467271ef31028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:46.994207 22405 tablet_service.cc:1467] Processing CreateTablet for tablet 519d76fb214542a4a9ad9c9cccfa35dc (DEFAULT_TABLE table=test-table [id=5a650c3db8f64bf8b29467271ef31028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:46.995574 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519d76fb214542a4a9ad9c9cccfa35dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:46.995860 22405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519d76fb214542a4a9ad9c9cccfa35dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:47.009975 22489 tablet_bootstrap.cc:492] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: Bootstrap starting.
I20260927 22:06:47.015720 22489 tablet_bootstrap.cc:654] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:47.017545 22489 log.cc:824] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:47.019671 22490 tablet_bootstrap.cc:492] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: Bootstrap starting.
I20260927 22:06:47.020109 22491 tablet_bootstrap.cc:492] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: Bootstrap starting.
I20260927 22:06:47.024758 22489 tablet_bootstrap.cc:492] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: No bootstrap required, opened a new log
I20260927 22:06:47.024796 22490 tablet_bootstrap.cc:654] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:47.025151 22489 ts_tablet_manager.cc:1397] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260927 22:06:47.025533 22491 tablet_bootstrap.cc:654] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:47.026475 22490 log.cc:824] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:47.027012 22491 log.cc:824] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:47.031374 22491 tablet_bootstrap.cc:492] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: No bootstrap required, opened a new log
I20260927 22:06:47.031827 22491 ts_tablet_manager.cc:1397] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20260927 22:06:47.031804 22490 tablet_bootstrap.cc:492] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: No bootstrap required, opened a new log
I20260927 22:06:47.032373 22490 ts_tablet_manager.cc:1397] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260927 22:06:47.038151 22489 raft_consensus.cc:348] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b092306be5fc4976b235b6a488c00a9d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 37049 } } peers { permanent_uuid: "333eccd9f57146349ad9371c6cb17694" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34787 } } peers { permanent_uuid: "a576d10a5fef42139f4b09ed87b5529f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34383 } }
I20260927 22:06:47.038760 22489 raft_consensus.cc:374] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:47.038887 22489 raft_consensus.cc:729] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333eccd9f57146349ad9371c6cb17694, State: Initialized, Role: FOLLOWER
I20260927 22:06:47.039676 22489 consensus_queue.cc:260] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b092306be5fc4976b235b6a488c00a9d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 37049 } } peers { permanent_uuid: "333eccd9f57146349ad9371c6cb17694" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34787 } } peers { permanent_uuid: "a576d10a5fef42139f4b09ed87b5529f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34383 } }
I20260927 22:06:47.042053 22489 ts_tablet_manager.cc:1428] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: Time spent starting tablet: real 0.017s user 0.019s sys 0.000s
I20260927 22:06:47.043458 22491 raft_consensus.cc:348] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b092306be5fc4976b235b6a488c00a9d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 37049 } } peers { permanent_uuid: "333eccd9f57146349ad9371c6cb17694" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34787 } } peers { permanent_uuid: "a576d10a5fef42139f4b09ed87b5529f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34383 } }
I20260927 22:06:47.044127 22491 raft_consensus.cc:374] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:47.044247 22491 raft_consensus.cc:729] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a576d10a5fef42139f4b09ed87b5529f, State: Initialized, Role: FOLLOWER
I20260927 22:06:47.045174 22491 consensus_queue.cc:260] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b092306be5fc4976b235b6a488c00a9d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 37049 } } peers { permanent_uuid: "333eccd9f57146349ad9371c6cb17694" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34787 } } peers { permanent_uuid: "a576d10a5fef42139f4b09ed87b5529f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34383 } }
I20260927 22:06:47.045185 22490 raft_consensus.cc:348] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b092306be5fc4976b235b6a488c00a9d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 37049 } } peers { permanent_uuid: "333eccd9f57146349ad9371c6cb17694" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34787 } } peers { permanent_uuid: "a576d10a5fef42139f4b09ed87b5529f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34383 } }
I20260927 22:06:47.045889 22490 raft_consensus.cc:374] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:47.046020 22490 raft_consensus.cc:729] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b092306be5fc4976b235b6a488c00a9d, State: Initialized, Role: FOLLOWER
I20260927 22:06:47.046921 22470 heartbeater.cc:503] Master 127.3.159.62:36389 was elected leader, sending a full tablet report...
I20260927 22:06:47.047055 22490 consensus_queue.cc:260] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b092306be5fc4976b235b6a488c00a9d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 37049 } } peers { permanent_uuid: "333eccd9f57146349ad9371c6cb17694" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34787 } } peers { permanent_uuid: "a576d10a5fef42139f4b09ed87b5529f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34383 } }
I20260927 22:06:47.049728 22490 ts_tablet_manager.cc:1428] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260927 22:06:47.050156 22491 ts_tablet_manager.cc:1428] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260927 22:06:47.052326 22336 heartbeater.cc:503] Master 127.3.159.62:36389 was elected leader, sending a full tablet report...
W20260927 22:06:47.111119 22471 tablet.cc:2369] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:06:47.127696 22337 tablet.cc:2369] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:06:47.199738 22203 tablet.cc:2369] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:06:47.385923 22496 raft_consensus.cc:482] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:06:47.386935 22496 raft_consensus.cc:504] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b092306be5fc4976b235b6a488c00a9d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 37049 } } peers { permanent_uuid: "333eccd9f57146349ad9371c6cb17694" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34787 } } peers { permanent_uuid: "a576d10a5fef42139f4b09ed87b5529f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34383 } }
I20260927 22:06:47.390586 22496 leader_election.cc:288] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b092306be5fc4976b235b6a488c00a9d (127.3.159.2:37049), 333eccd9f57146349ad9371c6cb17694 (127.3.159.1:34787)
I20260927 22:06:47.400677 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:47.390868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:57369 (local address 127.3.159.2:37049)
0927 22:06:47.391281 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:47.391290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:47.391431 (+ 141us) server_negotiation.cc:407] Connection header received
0927 22:06:47.391732 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:47.391737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:47.391830 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:47.391981 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:47.392617 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.394088 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.397201 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.399191 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.399510 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:47.399652 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:47.400197 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 22:06:47.400378 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 22:06:47.400802 22473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:47.390752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:37049 (local address 127.3.159.3:57369)
0927 22:06:47.391295 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:47.391326 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:47.391537 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:47.392035 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:47.392040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:47.392076 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:47.392421 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:47.392441 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.394281 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.394286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:47.396826 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:47.396854 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.399616 (+ 2762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.399623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:47.399897 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:47.399937 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 22:06:47.400131 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 22:06:47.400346 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":47}
I20260927 22:06:47.401126 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:47.392192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:47715 (local address 127.3.159.1:34787)
0927 22:06:47.392478 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:47.392485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:47.392918 (+ 433us) server_negotiation.cc:407] Connection header received
0927 22:06:47.393259 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:47.393270 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:47.393485 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:47.393656 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:47.394437 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.395614 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.398247 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.399920 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.400062 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:47.400212 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:47.400680 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 22:06:47.400863 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260927 22:06:47.401872 22498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:47.391836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:34787 (local address 127.3.159.3:47715)
0927 22:06:47.392766 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:47.392787 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:47.392928 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:47.393723 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:47.393727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:47.393754 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:47.394131 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:47.394141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.395895 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.395902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:47.397941 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:47.397961 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.400269 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.400276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:47.400452 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:47.400488 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 22:06:47.400609 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 22:06:47.401413 (+ 804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":1,"thread_start_us":458,"threads_started":1}
I20260927 22:06:47.402607 22291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519d76fb214542a4a9ad9c9cccfa35dc" candidate_uuid: "a576d10a5fef42139f4b09ed87b5529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b092306be5fc4976b235b6a488c00a9d" is_pre_election: true
I20260927 22:06:47.403487 22291 raft_consensus.cc:2454] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a576d10a5fef42139f4b09ed87b5529f in term 0.
I20260927 22:06:47.403698 22157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519d76fb214542a4a9ad9c9cccfa35dc" candidate_uuid: "a576d10a5fef42139f4b09ed87b5529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333eccd9f57146349ad9371c6cb17694" is_pre_election: true
I20260927 22:06:47.404412 22157 raft_consensus.cc:2454] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a576d10a5fef42139f4b09ed87b5529f in term 0.
I20260927 22:06:47.404592 22360 leader_election.cc:302] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a576d10a5fef42139f4b09ed87b5529f, b092306be5fc4976b235b6a488c00a9d; no voters:
I20260927 22:06:47.405304 22496 raft_consensus.cc:2781] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:06:47.405576 22496 raft_consensus.cc:482] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:06:47.405709 22496 raft_consensus.cc:3037] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:47.408941 22496 raft_consensus.cc:504] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b092306be5fc4976b235b6a488c00a9d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 37049 } } peers { permanent_uuid: "333eccd9f57146349ad9371c6cb17694" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34787 } } peers { permanent_uuid: "a576d10a5fef42139f4b09ed87b5529f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34383 } }
I20260927 22:06:47.410382 22496 leader_election.cc:288] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [CANDIDATE]: Term 1 election: Requested vote from peers b092306be5fc4976b235b6a488c00a9d (127.3.159.2:37049), 333eccd9f57146349ad9371c6cb17694 (127.3.159.1:34787)
I20260927 22:06:47.411026 22291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519d76fb214542a4a9ad9c9cccfa35dc" candidate_uuid: "a576d10a5fef42139f4b09ed87b5529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b092306be5fc4976b235b6a488c00a9d"
I20260927 22:06:47.411180 22157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519d76fb214542a4a9ad9c9cccfa35dc" candidate_uuid: "a576d10a5fef42139f4b09ed87b5529f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "333eccd9f57146349ad9371c6cb17694"
I20260927 22:06:47.411369 22291 raft_consensus.cc:3037] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:47.411587 22157 raft_consensus.cc:3037] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:47.414501 22291 raft_consensus.cc:2454] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a576d10a5fef42139f4b09ed87b5529f in term 1.
I20260927 22:06:47.415738 22360 leader_election.cc:302] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a576d10a5fef42139f4b09ed87b5529f, b092306be5fc4976b235b6a488c00a9d; no voters:
I20260927 22:06:47.416431 22496 raft_consensus.cc:2781] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:06:47.417261 22157 raft_consensus.cc:2454] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a576d10a5fef42139f4b09ed87b5529f in term 1.
I20260927 22:06:47.417328 22496 raft_consensus.cc:686] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 1 LEADER]: Becoming Leader. State: Replica: a576d10a5fef42139f4b09ed87b5529f, State: Running, Role: LEADER
I20260927 22:06:47.418394 22496 consensus_queue.cc:237] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b092306be5fc4976b235b6a488c00a9d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 37049 } } peers { permanent_uuid: "333eccd9f57146349ad9371c6cb17694" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34787 } } peers { permanent_uuid: "a576d10a5fef42139f4b09ed87b5529f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34383 } }
I20260927 22:06:47.425732 22013 catalog_manager.cc:5495] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f reported cstate change: term changed from 0 to 1, leader changed from <none> to a576d10a5fef42139f4b09ed87b5529f (127.3.159.3). New cstate: current_term: 1 leader_uuid: "a576d10a5fef42139f4b09ed87b5529f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b092306be5fc4976b235b6a488c00a9d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 37049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "333eccd9f57146349ad9371c6cb17694" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a576d10a5fef42139f4b09ed87b5529f" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 34383 } health_report { overall_health: HEALTHY } } }
I20260927 22:06:47.450467 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:47.441868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.3.159.3:34383)
0927 22:06:47.442189 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:47.442195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:47.442637 (+ 442us) server_negotiation.cc:407] Connection header received
0927 22:06:47.442896 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:47.442902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:47.443052 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:47.443245 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:06:47.443857 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.445113 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.447121 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.447793 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.447939 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:47.448666 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:06:47.449562 (+ 896us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:06:47.449764 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:47.450061 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 22:06:47.450126 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:06:47.462174 22157 raft_consensus.cc:1261] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694 [term 1 FOLLOWER]: Refusing update from remote peer a576d10a5fef42139f4b09ed87b5529f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:06:47.462482 22291 raft_consensus.cc:1261] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d [term 1 FOLLOWER]: Refusing update from remote peer a576d10a5fef42139f4b09ed87b5529f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:06:47.463653 22496 consensus_queue.cc:1035] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [LEADER]: Connected to new peer: Peer: permanent_uuid: "333eccd9f57146349ad9371c6cb17694" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 34787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:06:47.464243 22499 consensus_queue.cc:1035] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b092306be5fc4976b235b6a488c00a9d" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 37049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:06:47.513414 22012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:06:47.514206 22012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:06:47.520068 22012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fb770b202054162819b5643097b6734: Sending DeleteTablet for 3 replicas of tablet 519d76fb214542a4a9ad9c9cccfa35dc
I20260927 22:06:47.522266 22137 tablet_service.cc:1514] Processing DeleteTablet for tablet 519d76fb214542a4a9ad9c9cccfa35dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:47 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:55478
I20260927 22:06:47.522253 22271 tablet_service.cc:1514] Processing DeleteTablet for tablet 519d76fb214542a4a9ad9c9cccfa35dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:47 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:35418
I20260927 22:06:47.523104 22405 tablet_service.cc:1514] Processing DeleteTablet for tablet 519d76fb214542a4a9ad9c9cccfa35dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:47 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:56306
I20260927 22:06:47.526207 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 22074
I20260927 22:06:47.538240 22514 tablet_replica.cc:330] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: stopping tablet replica
I20260927 22:06:47.539057 22514 raft_consensus.cc:2229] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:06:47.539898 22514 raft_consensus.cc:2258] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:47.543900 22514 ts_tablet_manager.cc:1905] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:47.554500 22514 ts_tablet_manager.cc:1918] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:47.554761 22514 log.cc:1196] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/519d76fb214542a4a9ad9c9cccfa35dc
I20260927 22:06:47.555498 22514 ts_tablet_manager.cc:1939] T 519d76fb214542a4a9ad9c9cccfa35dc P b092306be5fc4976b235b6a488c00a9d: Deleting consensus metadata
I20260927 22:06:47.556967 22516 tablet_replica.cc:330] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: stopping tablet replica
I20260927 22:06:47.558223 21999 catalog_manager.cc:4841] TS b092306be5fc4976b235b6a488c00a9d (127.3.159.2:37049): tablet 519d76fb214542a4a9ad9c9cccfa35dc (table test-table [id=5a650c3db8f64bf8b29467271ef31028]) successfully deleted
I20260927 22:06:47.559127 22516 raft_consensus.cc:2229] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:06:47.560037 22516 raft_consensus.cc:2258] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:47.560801 22515 tablet_replica.cc:330] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: stopping tablet replica
I20260927 22:06:47.561569 22515 raft_consensus.cc:2229] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:06:47.562229 22515 raft_consensus.cc:2258] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:47.562871 22516 ts_tablet_manager.cc:1905] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:47.564925 22515 ts_tablet_manager.cc:1905] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 22:06:47 dist-test-slave-sg4j krb5kdc[21968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:06:47.573045 22516 ts_tablet_manager.cc:1918] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:47.573453 22516 log.cc:1196] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/519d76fb214542a4a9ad9c9cccfa35dc
I20260927 22:06:47.574239 22516 ts_tablet_manager.cc:1939] T 519d76fb214542a4a9ad9c9cccfa35dc P a576d10a5fef42139f4b09ed87b5529f: Deleting consensus metadata
I20260927 22:06:47.574820 22515 ts_tablet_manager.cc:1918] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:47.575047 22515 log.cc:1196] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/519d76fb214542a4a9ad9c9cccfa35dc
I20260927 22:06:47.575784 22515 ts_tablet_manager.cc:1939] T 519d76fb214542a4a9ad9c9cccfa35dc P 333eccd9f57146349ad9371c6cb17694: Deleting consensus metadata
I20260927 22:06:47.576922 21997 catalog_manager.cc:4841] TS a576d10a5fef42139f4b09ed87b5529f (127.3.159.3:34383): tablet 519d76fb214542a4a9ad9c9cccfa35dc (table test-table [id=5a650c3db8f64bf8b29467271ef31028]) successfully deleted
I20260927 22:06:47.578737 21997 catalog_manager.cc:4841] TS 333eccd9f57146349ad9371c6cb17694 (127.3.159.1:34787): tablet 519d76fb214542a4a9ad9c9cccfa35dc (table test-table [id=5a650c3db8f64bf8b29467271ef31028]) successfully deleted
I20260927 22:06:47.579481 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:47.528970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.3.159.1:34787)
0927 22:06:47.529287 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:47.529300 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:47.552863 (+ 23563us) server_negotiation.cc:407] Connection header received
0927 22:06:47.553584 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:47.553591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:47.553711 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:47.553790 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:47.557557 (+ 3767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.558807 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.560899 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.561627 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.561758 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:47.569143 (+ 7385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:47.569173 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:47.569199 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:47.569276 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:47.575212 (+ 5936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:47.576569 (+ 1357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:47.576582 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:47.576589 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:47.576878 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:47.577485 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:47.577493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:47.577496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:47.577965 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:47.578362 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:47.578700 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 22:06:47.579018 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 22:06:47.580826 22197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:47.700141 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 22074
I20260927 22:06:47.730068 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 22208
Sep 27 22:06:47 dist-test-slave-sg4j krb5kdc[21968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:06:47.755899 22484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:47.731702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.3.159.2:37049)
0927 22:06:47.731942 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:47.731950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:47.732034 (+ 84us) server_negotiation.cc:407] Connection header received
0927 22:06:47.732246 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:47.732253 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:47.732362 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:47.732699 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:47.734283 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.735612 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.737360 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.738165 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.738331 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:47.745872 (+ 7541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:47.745900 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:47.745925 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:47.745995 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:47.750651 (+ 4656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:47.751969 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:47.751981 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:47.751984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:47.752253 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:47.752847 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:47.752852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:47.752854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:47.753582 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:47.753910 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:47.754763 (+ 853us) server_negotiation.cc:299] Negotiation successful
0927 22:06:47.755361 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 22:06:47.756976 22331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:47.878363 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 22208
I20260927 22:06:47.899446 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 22342
Sep 27 22:06:47 dist-test-slave-sg4j krb5kdc[21968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:06:47.922948 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:47.900611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.3.159.3:34383)
0927 22:06:47.900901 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:47.900908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:47.900937 (+ 29us) server_negotiation.cc:407] Connection header received
0927 22:06:47.901032 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:47.901038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:47.901133 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:47.901300 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:47.903297 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.904472 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.906177 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:47.906898 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:47.907076 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:47.912686 (+ 5610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:47.912710 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:47.912733 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:47.912793 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:47.917997 (+ 5204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:47.919238 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:47.919254 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:47.919257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:47.919686 (+ 429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:47.920399 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:47.920405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:47.920409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:47.921081 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:47.921553 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:47.922145 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 22:06:47.922499 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260927 22:06:47.924135 22465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:48.075078 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 22342
I20260927 22:06:48.110116 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 21981
I20260927 22:06:48.112214 22043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:48.248513 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 21981
2026-09-27T22:06:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5765 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T22:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:06:48Z Disabled control of system clock
I20260927 22:06:48.353277 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:34275
--webserver_interface=127.3.159.62
--webserver_port=0
--builtin_ntp_servers=127.3.159.20:37197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:34275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:06:48.835825 22525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:48.836289 22525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:48.836460 22525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:48.847287 22525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:06:48.847494 22525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:48.847579 22525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:48.847683 22525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:06:48.847772 22525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:06:48.859311 22525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:37197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:34275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:34275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:48.861315 22525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:48.864042 22525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:48.881249 22530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:48.881268 22531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:48.881214 22533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:48.888401 22525 server_base.cc:1034] running on GCE node
I20260927 22:06:48.889935 22525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:48.892460 22525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:48.896009 22525 hybrid_clock.cc:648] HybridClock initialized: now 1790546808895882 us; error 112 us; skew 500 ppm
I20260927 22:06:48.896790 22525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:06:48.901095 22525 webserver.cc:460] Webserver started at http://127.3.159.62:33961/ using document root <none> and password file <none>
I20260927 22:06:48.902227 22525 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:48.902402 22525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:48.902978 22525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:48.909298 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "54f5410d4aca42d59c17b54ac4cb7aa8"
format_stamp: "Formatted at 2026-09-27 22:06:48 on dist-test-slave-sg4j"
server_key: "1d5f6f24d7c7f21b00ded0aecc97b0e2"
server_key_iv: "c42720f2023f735b12c70688d52720e4"
server_key_version: "encryptionkey@0"
I20260927 22:06:48.911170 22525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "54f5410d4aca42d59c17b54ac4cb7aa8"
format_stamp: "Formatted at 2026-09-27 22:06:48 on dist-test-slave-sg4j"
server_key: "1d5f6f24d7c7f21b00ded0aecc97b0e2"
server_key_iv: "c42720f2023f735b12c70688d52720e4"
server_key_version: "encryptionkey@0"
I20260927 22:06:48.921072 22525 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.005s sys 0.004s
I20260927 22:06:48.926901 22539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:48.930084 22525 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260927 22:06:48.930428 22525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "54f5410d4aca42d59c17b54ac4cb7aa8"
format_stamp: "Formatted at 2026-09-27 22:06:48 on dist-test-slave-sg4j"
server_key: "1d5f6f24d7c7f21b00ded0aecc97b0e2"
server_key_iv: "c42720f2023f735b12c70688d52720e4"
server_key_version: "encryptionkey@0"
I20260927 22:06:48.930850 22525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:48.974822 22525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:48.976125 22525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:48.976447 22525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:49.002377 22525 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:34275
I20260927 22:06:49.002406 22590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:34275 every 8 connection(s)
I20260927 22:06:49.004360 22525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:06:49.009529 22591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:49.014791 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 22525
I20260927 22:06:49.015050 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:06:49.015779 3708 external_mini_cluster.cc:1428] Setting key 3775450efdedd8312af4fa84e6bd9ac8
I20260927 22:06:49.025310 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8: Bootstrap starting.
I20260927 22:06:49.030223 22591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:49.031721 22591 log.cc:824] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:49.036751 22591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8: No bootstrap required, opened a new log
I20260927 22:06:49.037355 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:49.018650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51740 (local address 127.3.159.62:34275)
0927 22:06:49.020313 (+ 1663us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:49.020332 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:49.020404 (+ 72us) server_negotiation.cc:407] Connection header received
0927 22:06:49.022890 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:49.022931 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:49.023703 (+ 772us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:49.024199 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:49.025137 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:49.027059 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:49.029576 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:49.031193 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:49.031486 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:49.033263 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:49.033281 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:49.033318 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:49.034425 (+ 1107us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:49.034585 (+ 160us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:49.034621 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:49.034766 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:49.035206 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 22:06:49.035791 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":691,"threads_started":1}
I20260927 22:06:49.046372 22591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f5410d4aca42d59c17b54ac4cb7aa8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34275 } }
I20260927 22:06:49.046831 22591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:49.046902 22591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f5410d4aca42d59c17b54ac4cb7aa8, State: Initialized, Role: FOLLOWER
I20260927 22:06:49.047917 22591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f5410d4aca42d59c17b54ac4cb7aa8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34275 } }
I20260927 22:06:49.048218 22591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:06:49.048420 22591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:06:49.048799 22591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:49.051851 22591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f5410d4aca42d59c17b54ac4cb7aa8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34275 } }
I20260927 22:06:49.052949 22591 leader_election.cc:302] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54f5410d4aca42d59c17b54ac4cb7aa8; no voters:
I20260927 22:06:49.053933 22591 leader_election.cc:288] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:06:49.054207 22596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:06:49.055825 22596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [term 1 LEADER]: Becoming Leader. State: Replica: 54f5410d4aca42d59c17b54ac4cb7aa8, State: Running, Role: LEADER
I20260927 22:06:49.056591 22596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f5410d4aca42d59c17b54ac4cb7aa8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34275 } }
I20260927 22:06:49.056988 22591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:06:49.062062 22598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f5410d4aca42d59c17b54ac4cb7aa8. Latest consensus state: current_term: 1 leader_uuid: "54f5410d4aca42d59c17b54ac4cb7aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f5410d4aca42d59c17b54ac4cb7aa8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34275 } } }
I20260927 22:06:49.062047 22597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54f5410d4aca42d59c17b54ac4cb7aa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f5410d4aca42d59c17b54ac4cb7aa8" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34275 } } }
I20260927 22:06:49.063494 22598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:49.063956 22597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8 [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:49.065704 22606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:06:49.076349 22606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:06:49.089159 22606 catalog_manager.cc:1347] Generated new cluster ID: 025cae2991494887b1bf9009f3fc1a27
I20260927 22:06:49.089314 22606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:06:49.116669 22606 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:06:49.118943 22606 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:06:49.134514 22606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8: Generated new TSK 0
I20260927 22:06:49.135824 22606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:06:49.153928 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34275
--builtin_ntp_servers=127.3.159.20:37197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:06:49.632692 22615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:49.633064 22615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:49.633240 22615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:49.641535 22615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:49.641664 22615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:49.641844 22615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:06:49.653019 22615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:37197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:49.654549 22615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:49.656522 22615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:49.672719 22623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:49.672847 22621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:49.672745 22620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:49.674973 22615 server_base.cc:1034] running on GCE node
I20260927 22:06:49.676026 22615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:49.677678 22615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:49.679006 22615 hybrid_clock.cc:648] HybridClock initialized: now 1790546809678895 us; error 90 us; skew 500 ppm
I20260927 22:06:49.679571 22615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:06:49.683928 22615 webserver.cc:460] Webserver started at http://127.3.159.1:37951/ using document root <none> and password file <none>
I20260927 22:06:49.685034 22615 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:49.685212 22615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:49.685699 22615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:49.689613 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "5641383881414581a29d2ed15ba0fc36"
format_stamp: "Formatted at 2026-09-27 22:06:49 on dist-test-slave-sg4j"
server_key: "52b46ca85c6c89f198941f6257ae4021"
server_key_iv: "c22d055fbb4059b0fe8e76b403bbf0a1"
server_key_version: "encryptionkey@0"
I20260927 22:06:49.690579 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "5641383881414581a29d2ed15ba0fc36"
format_stamp: "Formatted at 2026-09-27 22:06:49 on dist-test-slave-sg4j"
server_key: "52b46ca85c6c89f198941f6257ae4021"
server_key_iv: "c22d055fbb4059b0fe8e76b403bbf0a1"
server_key_version: "encryptionkey@0"
I20260927 22:06:49.698019 22615 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:06:49.702963 22629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:49.705132 22615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260927 22:06:49.705508 22615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "5641383881414581a29d2ed15ba0fc36"
format_stamp: "Formatted at 2026-09-27 22:06:49 on dist-test-slave-sg4j"
server_key: "52b46ca85c6c89f198941f6257ae4021"
server_key_iv: "c22d055fbb4059b0fe8e76b403bbf0a1"
server_key_version: "encryptionkey@0"
I20260927 22:06:49.705909 22615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:49.730232 22615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:49.731487 22615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:49.731902 22615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:49.733516 22615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:49.735780 22615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:49.735941 22615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:49.736104 22615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:49.736202 22615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:49.783113 22615 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:35525
I20260927 22:06:49.783259 22741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:35525 every 8 connection(s)
I20260927 22:06:49.784919 22615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:06:49.787982 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 22615
I20260927 22:06:49.788197 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:06:49.788798 3708 external_mini_cluster.cc:1428] Setting key 789e46827646a3dbb2be35487d846a0b
I20260927 22:06:49.792564 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34275
--builtin_ntp_servers=127.3.159.20:37197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:06:49.802999 22744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:49.788701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:60727 (local address 127.3.159.62:34275)
0927 22:06:49.789320 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:49.789325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:49.791523 (+ 2198us) server_negotiation.cc:407] Connection header received
0927 22:06:49.793726 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:49.793737 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:49.793862 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:49.794034 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:49.795485 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:49.796723 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:49.798847 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:49.799744 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:49.799917 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:49.801722 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:49.801731 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:49.801739 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:49.801776 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:49.801847 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:49.801867 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:49.801986 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:49.802292 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 22:06:49.802459 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":378,"threads_started":1}
I20260927 22:06:49.804193 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:49.789637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:34275 (local address 127.3.159.1:60727)
0927 22:06:49.791214 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:49.791277 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:49.793300 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:49.794377 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:49.794388 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:49.794507 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:49.795238 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:49.795268 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:49.796989 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:49.796994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:49.798512 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:49.798531 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:49.800071 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:49.800078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:49.800348 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:49.801231 (+ 883us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:06:49.801234 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:49.801283 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:06:49.801458 (+ 175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:49.802020 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:49.802032 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:49.802039 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 22:06:49.802227 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 22:06:49.802621 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":456,"threads_started":1}
I20260927 22:06:49.806384 22742 heartbeater.cc:347] Connected to a master server at 127.3.159.62:34275
I20260927 22:06:49.806988 22742 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:49.808178 22742 heartbeater.cc:511] Master 127.3.159.62:34275 requested a full tablet report, sending...
I20260927 22:06:49.811380 22556 ts_manager.cc:194] Registered new tserver with Master: 5641383881414581a29d2ed15ba0fc36 (127.3.159.1:35525)
I20260927 22:06:49.813261 22556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.1:60727
W20260927 22:06:50.228371 22746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:50.228979 22746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:50.229198 22746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:50.240716 22746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:50.241144 22746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:50.241453 22746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:06:50.253882 22746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:37197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:50.255934 22746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:50.259668 22746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:50.280989 22752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:50.281209 22751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:50.281131 22754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:50.282146 22746 server_base.cc:1034] running on GCE node
I20260927 22:06:50.283288 22746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:50.284811 22746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:50.286286 22746 hybrid_clock.cc:648] HybridClock initialized: now 1790546810286158 us; error 90 us; skew 500 ppm
I20260927 22:06:50.286768 22746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:06:50.290232 22746 webserver.cc:460] Webserver started at http://127.3.159.2:36349/ using document root <none> and password file <none>
I20260927 22:06:50.291074 22746 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:50.291182 22746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:50.291785 22746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:50.295342 22746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "ab41ec4d809d4504ade878897d60e467"
format_stamp: "Formatted at 2026-09-27 22:06:50 on dist-test-slave-sg4j"
server_key: "abf6cfdefdc465fa3899730b6f4e17cc"
server_key_iv: "e29acaae61164ece0f7f960b6e789b04"
server_key_version: "encryptionkey@0"
I20260927 22:06:50.296262 22746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "ab41ec4d809d4504ade878897d60e467"
format_stamp: "Formatted at 2026-09-27 22:06:50 on dist-test-slave-sg4j"
server_key: "abf6cfdefdc465fa3899730b6f4e17cc"
server_key_iv: "e29acaae61164ece0f7f960b6e789b04"
server_key_version: "encryptionkey@0"
I20260927 22:06:50.303983 22746 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260927 22:06:50.309259 22760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:50.311625 22746 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260927 22:06:50.311869 22746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "ab41ec4d809d4504ade878897d60e467"
format_stamp: "Formatted at 2026-09-27 22:06:50 on dist-test-slave-sg4j"
server_key: "abf6cfdefdc465fa3899730b6f4e17cc"
server_key_iv: "e29acaae61164ece0f7f960b6e789b04"
server_key_version: "encryptionkey@0"
I20260927 22:06:50.312131 22746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:50.334204 22746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:50.335443 22746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:50.335740 22746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:50.337196 22746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:50.339600 22746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:50.339712 22746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:50.339802 22746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:50.339864 22746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:50.396113 22746 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:35983
I20260927 22:06:50.396309 22872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:35983 every 8 connection(s)
I20260927 22:06:50.398172 22746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:06:50.401664 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 22746
I20260927 22:06:50.401903 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:06:50.402695 3708 external_mini_cluster.cc:1428] Setting key 81dce5f4d7ee4fd012b3592145643de6
I20260927 22:06:50.407748 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34275
--builtin_ntp_servers=127.3.159.20:37197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:06:50.429150 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:50.402448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:49757 (local address 127.3.159.62:34275)
0927 22:06:50.403287 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:50.403306 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:50.405147 (+ 1841us) server_negotiation.cc:407] Connection header received
0927 22:06:50.407659 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:50.407667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:50.407809 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:50.408010 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:50.410385 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:50.411667 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:50.417134 (+ 5467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:50.418065 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:50.418281 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:50.427034 (+ 8753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:50.427046 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:50.427055 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:50.427098 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:50.427187 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:50.427214 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:50.427362 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:50.428402 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0927 22:06:50.428594 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":366,"threads_started":1}
I20260927 22:06:50.431186 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:50.403342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:34275 (local address 127.3.159.2:49757)
0927 22:06:50.404902 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:50.404970 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:50.407175 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:50.408646 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:50.408667 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:50.408850 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:50.409971 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:50.410010 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:50.412016 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:50.412026 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:50.416648 (+ 4622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:50.416693 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:50.418685 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:50.418706 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:50.419487 (+ 781us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:50.426283 (+ 6796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:06:50.426290 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:50.426352 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:06:50.426519 (+ 167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:50.427869 (+ 1350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:50.427892 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:50.427903 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:06:50.428294 (+ 391us) client_negotiation.cc:240] Negotiation successful
0927 22:06:50.429041 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":485,"threads_started":1}
I20260927 22:06:50.434121 22873 heartbeater.cc:347] Connected to a master server at 127.3.159.62:34275
I20260927 22:06:50.434706 22873 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:50.436208 22873 heartbeater.cc:511] Master 127.3.159.62:34275 requested a full tablet report, sending...
I20260927 22:06:50.439265 22556 ts_manager.cc:194] Registered new tserver with Master: ab41ec4d809d4504ade878897d60e467 (127.3.159.2:35983)
I20260927 22:06:50.440876 22556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.2:49757
I20260927 22:06:50.817899 22742 heartbeater.cc:503] Master 127.3.159.62:34275 was elected leader, sending a full tablet report...
W20260927 22:06:50.868074 22877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:50.868533 22877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:50.868724 22877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:50.878628 22877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:50.878778 22877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:50.878983 22877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:06:50.890293 22877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:37197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34275
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:50.892206 22877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:50.894724 22877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:50.926913 22882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:50.926800 22883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:50.928337 22885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:50.930626 22877 server_base.cc:1034] running on GCE node
I20260927 22:06:50.931690 22877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:50.933269 22877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:50.934747 22877 hybrid_clock.cc:648] HybridClock initialized: now 1790546810934633 us; error 85 us; skew 500 ppm
I20260927 22:06:50.935321 22877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:06:50.939834 22877 webserver.cc:460] Webserver started at http://127.3.159.3:40721/ using document root <none> and password file <none>
I20260927 22:06:50.940979 22877 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:50.941180 22877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:50.941715 22877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:50.946213 22877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "e9d7175be4514a36b2ac4d5b4b97a876"
format_stamp: "Formatted at 2026-09-27 22:06:50 on dist-test-slave-sg4j"
server_key: "4f598fbf55a97c2d1dc9b2a537ab34ca"
server_key_iv: "6f6577dc342f11be768897dc9f03e437"
server_key_version: "encryptionkey@0"
I20260927 22:06:50.947386 22877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "e9d7175be4514a36b2ac4d5b4b97a876"
format_stamp: "Formatted at 2026-09-27 22:06:50 on dist-test-slave-sg4j"
server_key: "4f598fbf55a97c2d1dc9b2a537ab34ca"
server_key_iv: "6f6577dc342f11be768897dc9f03e437"
server_key_version: "encryptionkey@0"
I20260927 22:06:50.955536 22877 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260927 22:06:50.961247 22891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:50.964090 22877 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260927 22:06:50.964387 22877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "e9d7175be4514a36b2ac4d5b4b97a876"
format_stamp: "Formatted at 2026-09-27 22:06:50 on dist-test-slave-sg4j"
server_key: "4f598fbf55a97c2d1dc9b2a537ab34ca"
server_key_iv: "6f6577dc342f11be768897dc9f03e437"
server_key_version: "encryptionkey@0"
I20260927 22:06:50.964727 22877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:50.988511 22877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:50.989883 22877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:50.990253 22877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:50.991628 22877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:50.994133 22877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:50.994304 22877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:50.994479 22877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:50.994597 22877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:51.042521 22877 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:33255
I20260927 22:06:51.042589 23003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:33255 every 8 connection(s)
I20260927 22:06:51.044379 22877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:06:51.047214 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 22877
I20260927 22:06:51.047520 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:06:51.048264 3708 external_mini_cluster.cc:1428] Setting key 6573a5957f83560737e3988f1d811ee0
I20260927 22:06:51.064380 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.049250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:43433 (local address 127.3.159.62:34275)
0927 22:06:51.050002 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:51.050011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:51.052474 (+ 2463us) server_negotiation.cc:407] Connection header received
0927 22:06:51.054695 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:51.054700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:51.054799 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:51.054937 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:51.056373 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.057217 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.059269 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.059773 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.059897 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.062251 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:51.062257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:51.062261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:51.062291 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:51.062382 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:51.062408 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:51.062532 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:51.063131 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 22:06:51.063296 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":396,"threads_started":1}
I20260927 22:06:51.066327 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.050404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:34275 (local address 127.3.159.3:43433)
0927 22:06:51.052247 (+ 1843us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.052328 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:51.054328 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:51.055309 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:51.055319 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:51.055428 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:51.056155 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.056173 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.057482 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.057490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.059066 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.059083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.060045 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.060051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.060320 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.061761 (+ 1441us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:06:51.061764 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:51.061816 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:06:51.061932 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:51.062706 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:51.062718 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:51.062726 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 22:06:51.063064 (+ 338us) client_negotiation.cc:240] Negotiation successful
0927 22:06:51.063566 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"thread_start_us":502,"threads_started":1}
I20260927 22:06:51.068836 23004 heartbeater.cc:347] Connected to a master server at 127.3.159.62:34275
I20260927 22:06:51.069365 23004 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:51.070616 23004 heartbeater.cc:511] Master 127.3.159.62:34275 requested a full tablet report, sending...
I20260927 22:06:51.072898 22556 ts_manager.cc:194] Registered new tserver with Master: e9d7175be4514a36b2ac4d5b4b97a876 (127.3.159.3:33255)
I20260927 22:06:51.074088 22556 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.3:43433
I20260927 22:06:51.079306 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:06:51.094084 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.085820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.3.159.62:34275)
0927 22:06:51.086084 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:51.086089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:51.086532 (+ 443us) server_negotiation.cc:407] Connection header received
0927 22:06:51.086706 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:51.086710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:51.086820 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:51.086960 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:51.087579 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.088498 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.090268 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.090927 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.091273 (+ 346us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.092858 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:51.092864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:51.092894 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:51.092922 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:51.092976 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:51.092992 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:51.093147 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:51.093678 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 22:06:51.093755 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:06:51.099623 22556 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:06:51.108021 22556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:06:51.138690 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.126856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:35983 (local address 127.0.0.1:32922)
0927 22:06:51.128517 (+ 1661us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.128535 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:51.128645 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:51.129795 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:51.129803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:51.129834 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:51.130338 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.130356 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.132399 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.132406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.134791 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.134805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.137480 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.137489 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.137816 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.137894 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 22:06:51.138058 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 22:06:51.138308 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1545,"mutex_wait_us":1,"thread_start_us":1385,"threads_started":1}
I20260927 22:06:51.140316 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.127270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.3.159.2:35983)
0927 22:06:51.128015 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:51.128024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:51.128821 (+ 797us) server_negotiation.cc:407] Connection header received
0927 22:06:51.128932 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:51.128940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:51.129234 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:51.129524 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:51.130544 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.132200 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.135026 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.136778 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.136945 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.137237 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:51.139573 (+ 2336us) server_negotiation.cc:299] Negotiation successful
0927 22:06:51.139836 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":369,"threads_started":1}
I20260927 22:06:51.141983 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.126304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.3.159.3:33255)
0927 22:06:51.127347 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:51.127359 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:51.128542 (+ 1183us) server_negotiation.cc:407] Connection header received
0927 22:06:51.130516 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:51.130525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:51.130793 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:51.130989 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:51.131966 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.133623 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.139287 (+ 5664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.140887 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.141051 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.141358 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:51.141469 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 22:06:51.141636 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":680,"threads_started":1}
I20260927 22:06:51.142513 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.126297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:33255 (local address 127.0.0.1:58670)
0927 22:06:51.128267 (+ 1970us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.128346 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:51.128590 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:51.131294 (+ 2704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:51.131299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:51.131327 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:51.131735 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.131748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.135303 (+ 3555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.135308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.139001 (+ 3693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.139017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.141084 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.141090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.141259 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.141291 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 22:06:51.141376 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 22:06:51.141549 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"mutex_wait_us":1332,"thread_start_us":1107,"threads_started":1}
I20260927 22:06:51.145238 22808 tablet_service.cc:1467] Processing CreateTablet for tablet 341795a9d1bd4a31ad56fcb2ad738738 (DEFAULT_TABLE table=test-table [id=8a71c20ed6bc4f89967f463f6f1f4ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:51.146986 22808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341795a9d1bd4a31ad56fcb2ad738738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:51.147199 22939 tablet_service.cc:1467] Processing CreateTablet for tablet 341795a9d1bd4a31ad56fcb2ad738738 (DEFAULT_TABLE table=test-table [id=8a71c20ed6bc4f89967f463f6f1f4ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:51.148872 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.129071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.3.159.1:35525)
0927 22:06:51.139361 (+ 10290us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:51.139385 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:51.139423 (+ 38us) server_negotiation.cc:407] Connection header received
0927 22:06:51.139550 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:51.139560 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:51.139912 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:51.140170 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:51.140740 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.142406 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.145051 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.146652 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.146819 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.147128 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:51.148172 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0927 22:06:51.148438 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9591,"thread_start_us":9419,"threads_started":1}
I20260927 22:06:51.149130 22939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341795a9d1bd4a31ad56fcb2ad738738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:51.148667 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.128656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35525 (local address 127.0.0.1:46480)
0927 22:06:51.138173 (+ 9517us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.138193 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:51.139262 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:51.140173 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:51.140179 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:51.140212 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:51.140542 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.140554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.142615 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.142622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.144712 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.144738 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.147295 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.147302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.147490 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.147527 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 22:06:51.147900 (+ 373us) client_negotiation.cc:240] Negotiation successful
0927 22:06:51.148067 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9297,"thread_start_us":9143,"threads_started":1}
I20260927 22:06:51.154699 22677 tablet_service.cc:1467] Processing CreateTablet for tablet 341795a9d1bd4a31ad56fcb2ad738738 (DEFAULT_TABLE table=test-table [id=8a71c20ed6bc4f89967f463f6f1f4ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:51.156580 22677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341795a9d1bd4a31ad56fcb2ad738738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:51.165318 23023 tablet_bootstrap.cc:492] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: Bootstrap starting.
I20260927 22:06:51.168534 23024 tablet_bootstrap.cc:492] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: Bootstrap starting.
I20260927 22:06:51.171489 23023 tablet_bootstrap.cc:654] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:51.173262 23023 log.cc:824] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:51.173964 23024 tablet_bootstrap.cc:654] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:51.174541 23025 tablet_bootstrap.cc:492] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: Bootstrap starting.
I20260927 22:06:51.176077 23024 log.cc:824] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:51.179083 23025 tablet_bootstrap.cc:654] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:51.181113 23023 tablet_bootstrap.cc:492] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: No bootstrap required, opened a new log
I20260927 22:06:51.181111 23024 tablet_bootstrap.cc:492] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: No bootstrap required, opened a new log
I20260927 22:06:51.181597 23024 ts_tablet_manager.cc:1397] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20260927 22:06:51.181630 23023 ts_tablet_manager.cc:1397] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20260927 22:06:51.180912 23025 log.cc:824] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:51.184990 23025 tablet_bootstrap.cc:492] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: No bootstrap required, opened a new log
I20260927 22:06:51.185343 23025 ts_tablet_manager.cc:1397] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260927 22:06:51.193185 23025 raft_consensus.cc:348] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33255 } } peers { permanent_uuid: "ab41ec4d809d4504ade878897d60e467" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35983 } } peers { permanent_uuid: "5641383881414581a29d2ed15ba0fc36" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35525 } }
I20260927 22:06:51.193773 23025 raft_consensus.cc:374] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:51.193872 23025 raft_consensus.cc:729] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5641383881414581a29d2ed15ba0fc36, State: Initialized, Role: FOLLOWER
I20260927 22:06:51.194233 23024 raft_consensus.cc:348] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33255 } } peers { permanent_uuid: "ab41ec4d809d4504ade878897d60e467" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35983 } } peers { permanent_uuid: "5641383881414581a29d2ed15ba0fc36" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35525 } }
I20260927 22:06:51.194233 23023 raft_consensus.cc:348] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33255 } } peers { permanent_uuid: "ab41ec4d809d4504ade878897d60e467" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35983 } } peers { permanent_uuid: "5641383881414581a29d2ed15ba0fc36" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35525 } }
I20260927 22:06:51.194841 23024 raft_consensus.cc:374] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:51.194847 23023 raft_consensus.cc:374] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:51.194963 23023 raft_consensus.cc:729] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d7175be4514a36b2ac4d5b4b97a876, State: Initialized, Role: FOLLOWER
I20260927 22:06:51.194963 23024 raft_consensus.cc:729] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab41ec4d809d4504ade878897d60e467, State: Initialized, Role: FOLLOWER
I20260927 22:06:51.195482 23025 consensus_queue.cc:260] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33255 } } peers { permanent_uuid: "ab41ec4d809d4504ade878897d60e467" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35983 } } peers { permanent_uuid: "5641383881414581a29d2ed15ba0fc36" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35525 } }
I20260927 22:06:51.195782 23023 consensus_queue.cc:260] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33255 } } peers { permanent_uuid: "ab41ec4d809d4504ade878897d60e467" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35983 } } peers { permanent_uuid: "5641383881414581a29d2ed15ba0fc36" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35525 } }
I20260927 22:06:51.195782 23024 consensus_queue.cc:260] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33255 } } peers { permanent_uuid: "ab41ec4d809d4504ade878897d60e467" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35983 } } peers { permanent_uuid: "5641383881414581a29d2ed15ba0fc36" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35525 } }
I20260927 22:06:51.197751 23025 ts_tablet_manager.cc:1428] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260927 22:06:51.199277 22873 heartbeater.cc:503] Master 127.3.159.62:34275 was elected leader, sending a full tablet report...
I20260927 22:06:51.199805 23004 heartbeater.cc:503] Master 127.3.159.62:34275 was elected leader, sending a full tablet report...
I20260927 22:06:51.200702 23023 ts_tablet_manager.cc:1428] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260927 22:06:51.202387 23024 ts_tablet_manager.cc:1428] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: Time spent starting tablet: real 0.021s user 0.013s sys 0.005s
W20260927 22:06:51.288506 22743 tablet.cc:2369] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:06:51.296999 23005 tablet.cc:2369] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:06:51.341624 23031 raft_consensus.cc:482] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:06:51.342020 23031 raft_consensus.cc:504] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33255 } } peers { permanent_uuid: "ab41ec4d809d4504ade878897d60e467" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35983 } } peers { permanent_uuid: "5641383881414581a29d2ed15ba0fc36" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35525 } }
I20260927 22:06:51.343969 23031 leader_election.cc:288] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab41ec4d809d4504ade878897d60e467 (127.3.159.2:35983), 5641383881414581a29d2ed15ba0fc36 (127.3.159.1:35525)
I20260927 22:06:51.351866 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.345667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:36433 (local address 127.3.159.1:35525)
0927 22:06:51.345907 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:51.345914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:51.345935 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:06:51.346012 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:51.346016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:51.346089 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:51.346198 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:51.346786 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.347542 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.350067 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.350876 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.351022 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.351138 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:51.351444 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 22:06:51.351573 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 22:06:51.351851 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.344712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:39039 (local address 127.3.159.2:35983)
0927 22:06:51.345363 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:51.345372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:51.345424 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:06:51.345519 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:51.345524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:51.345609 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:51.345741 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:51.346255 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.347111 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.349842 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.350712 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.350877 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.351001 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:51.351347 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 22:06:51.351500 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354}
I20260927 22:06:51.351903 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.344223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:35983 (local address 127.3.159.3:39039)
0927 22:06:51.344604 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.344659 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:51.345078 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:51.345813 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:51.345818 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:51.345851 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:51.346073 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.346084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.347279 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.347285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.349611 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.349632 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.350919 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.350926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.351158 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.351186 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 22:06:51.351311 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 22:06:51.351497 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":69}
I20260927 22:06:51.352022 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.344714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35525 (local address 127.3.159.3:36433)
0927 22:06:51.345572 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.345594 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:51.345780 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:51.346222 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:51.346227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:51.346255 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:51.346629 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.346639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.347711 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.347715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.349867 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:51.349883 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.351075 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.351087 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:51.351265 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.351294 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 22:06:51.351405 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 22:06:51.351564 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":2,"thread_start_us":426,"threads_started":1}
I20260927 22:06:51.353871 22697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341795a9d1bd4a31ad56fcb2ad738738" candidate_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5641383881414581a29d2ed15ba0fc36" is_pre_election: true
I20260927 22:06:51.353866 22828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341795a9d1bd4a31ad56fcb2ad738738" candidate_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab41ec4d809d4504ade878897d60e467" is_pre_election: true
I20260927 22:06:51.354791 22828 raft_consensus.cc:2454] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d7175be4514a36b2ac4d5b4b97a876 in term 0.
I20260927 22:06:51.354808 22697 raft_consensus.cc:2454] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d7175be4514a36b2ac4d5b4b97a876 in term 0.
I20260927 22:06:51.356086 22894 leader_election.cc:302] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab41ec4d809d4504ade878897d60e467, e9d7175be4514a36b2ac4d5b4b97a876; no voters:
I20260927 22:06:51.356658 23031 raft_consensus.cc:2781] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:06:51.356945 23031 raft_consensus.cc:482] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:06:51.357065 23031 raft_consensus.cc:3037] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:51.359941 23031 raft_consensus.cc:504] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33255 } } peers { permanent_uuid: "ab41ec4d809d4504ade878897d60e467" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35983 } } peers { permanent_uuid: "5641383881414581a29d2ed15ba0fc36" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35525 } }
I20260927 22:06:51.360632 23031 leader_election.cc:288] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [CANDIDATE]: Term 1 election: Requested vote from peers ab41ec4d809d4504ade878897d60e467 (127.3.159.2:35983), 5641383881414581a29d2ed15ba0fc36 (127.3.159.1:35525)
I20260927 22:06:51.361793 22697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341795a9d1bd4a31ad56fcb2ad738738" candidate_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5641383881414581a29d2ed15ba0fc36"
I20260927 22:06:51.361776 22828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341795a9d1bd4a31ad56fcb2ad738738" candidate_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab41ec4d809d4504ade878897d60e467"
I20260927 22:06:51.362159 22828 raft_consensus.cc:3037] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:51.362181 22697 raft_consensus.cc:3037] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:51.366559 22697 raft_consensus.cc:2454] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d7175be4514a36b2ac4d5b4b97a876 in term 1.
I20260927 22:06:51.366559 22828 raft_consensus.cc:2454] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d7175be4514a36b2ac4d5b4b97a876 in term 1.
I20260927 22:06:51.367534 22894 leader_election.cc:302] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab41ec4d809d4504ade878897d60e467, e9d7175be4514a36b2ac4d5b4b97a876; no voters:
I20260927 22:06:51.368113 23031 raft_consensus.cc:2781] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:06:51.368814 23031 raft_consensus.cc:686] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 1 LEADER]: Becoming Leader. State: Replica: e9d7175be4514a36b2ac4d5b4b97a876, State: Running, Role: LEADER
I20260927 22:06:51.370180 23031 consensus_queue.cc:237] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33255 } } peers { permanent_uuid: "ab41ec4d809d4504ade878897d60e467" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35983 } } peers { permanent_uuid: "5641383881414581a29d2ed15ba0fc36" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35525 } }
I20260927 22:06:51.376969 22555 catalog_manager.cc:5495] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9d7175be4514a36b2ac4d5b4b97a876 (127.3.159.3). New cstate: current_term: 1 leader_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d7175be4514a36b2ac4d5b4b97a876" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 33255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab41ec4d809d4504ade878897d60e467" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5641383881414581a29d2ed15ba0fc36" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35525 } health_report { overall_health: UNKNOWN } } }
W20260927 22:06:51.401702 22874 tablet.cc:2369] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:06:51.451905 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.444383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.3.159.3:33255)
0927 22:06:51.444678 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:51.444684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:51.444711 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:06:51.444841 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:51.444845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:51.444949 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:51.445133 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:06:51.445855 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.446840 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.448639 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.449255 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.449510 (+ 255us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.450298 (+ 788us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:06:51.451020 (+ 722us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:06:51.451199 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:51.451424 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 22:06:51.451556 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 22:06:51.463867 22828 raft_consensus.cc:1261] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467 [term 1 FOLLOWER]: Refusing update from remote peer e9d7175be4514a36b2ac4d5b4b97a876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:06:51.464089 22697 raft_consensus.cc:1261] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36 [term 1 FOLLOWER]: Refusing update from remote peer e9d7175be4514a36b2ac4d5b4b97a876: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:06:51.465308 23031 consensus_queue.cc:1035] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab41ec4d809d4504ade878897d60e467" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 35983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:06:51.465900 23033 consensus_queue.cc:1035] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5641383881414581a29d2ed15ba0fc36" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 35525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:06:51.505127 22556 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:06:51.505688 22556 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:06:51.511536 22556 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54f5410d4aca42d59c17b54ac4cb7aa8: Sending DeleteTablet for 3 replicas of tablet 341795a9d1bd4a31ad56fcb2ad738738
I20260927 22:06:51.513557 22939 tablet_service.cc:1514] Processing DeleteTablet for tablet 341795a9d1bd4a31ad56fcb2ad738738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:51 UTC) from {username='slave'} at 127.0.0.1:58670
I20260927 22:06:51.513607 22808 tablet_service.cc:1514] Processing DeleteTablet for tablet 341795a9d1bd4a31ad56fcb2ad738738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:51 UTC) from {username='slave'} at 127.0.0.1:32922
I20260927 22:06:51.514024 22677 tablet_service.cc:1514] Processing DeleteTablet for tablet 341795a9d1bd4a31ad56fcb2ad738738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:51 UTC) from {username='slave'} at 127.0.0.1:46480
I20260927 22:06:51.518534 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 22615
I20260927 22:06:51.526024 23048 tablet_replica.cc:330] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: stopping tablet replica
I20260927 22:06:51.526855 23048 raft_consensus.cc:2229] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:06:51.528263 23047 tablet_replica.cc:330] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: stopping tablet replica
I20260927 22:06:51.528896 23046 tablet_replica.cc:330] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: stopping tablet replica
I20260927 22:06:51.529258 23048 raft_consensus.cc:2258] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:51.529454 23046 raft_consensus.cc:2229] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:06:51.529991 23046 raft_consensus.cc:2258] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:51.532305 23048 ts_tablet_manager.cc:1905] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:51.532300 23046 ts_tablet_manager.cc:1905] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:51.532830 23047 raft_consensus.cc:2229] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:06:51.533768 23047 raft_consensus.cc:2258] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:51.540060 23047 ts_tablet_manager.cc:1905] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:51.540495 23046 ts_tablet_manager.cc:1918] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:51.540699 23046 log.cc:1196] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/341795a9d1bd4a31ad56fcb2ad738738
I20260927 22:06:51.541280 23046 ts_tablet_manager.cc:1939] T 341795a9d1bd4a31ad56fcb2ad738738 P ab41ec4d809d4504ade878897d60e467: Deleting consensus metadata
I20260927 22:06:51.543524 22542 catalog_manager.cc:4841] TS ab41ec4d809d4504ade878897d60e467 (127.3.159.2:35983): tablet 341795a9d1bd4a31ad56fcb2ad738738 (table test-table [id=8a71c20ed6bc4f89967f463f6f1f4ccd]) successfully deleted
I20260927 22:06:51.546434 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.520606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.3.159.1:35525)
0927 22:06:51.520855 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:51.520862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:51.533430 (+ 12568us) server_negotiation.cc:407] Connection header received
0927 22:06:51.534235 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:51.534243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:51.534401 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:51.534520 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:51.535414 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.537157 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.540548 (+ 3391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.541333 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.541498 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.543745 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:51.543769 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:51.543795 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:51.544981 (+ 1186us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:51.545096 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:51.545135 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:51.545277 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:51.545697 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 22:06:51.545881 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 22:06:51.548157 22737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:51.553363 23047 ts_tablet_manager.cc:1918] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:51.554075 23047 log.cc:1196] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/341795a9d1bd4a31ad56fcb2ad738738
I20260927 22:06:51.554713 23047 ts_tablet_manager.cc:1939] T 341795a9d1bd4a31ad56fcb2ad738738 P e9d7175be4514a36b2ac4d5b4b97a876: Deleting consensus metadata
I20260927 22:06:51.556797 22541 catalog_manager.cc:4841] TS e9d7175be4514a36b2ac4d5b4b97a876 (127.3.159.3:33255): tablet 341795a9d1bd4a31ad56fcb2ad738738 (table test-table [id=8a71c20ed6bc4f89967f463f6f1f4ccd]) successfully deleted
I20260927 22:06:51.655531 23048 ts_tablet_manager.cc:1918] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:51.655861 23048 log.cc:1196] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/341795a9d1bd4a31ad56fcb2ad738738
I20260927 22:06:51.656587 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 22615
I20260927 22:06:51.656632 23048 ts_tablet_manager.cc:1939] T 341795a9d1bd4a31ad56fcb2ad738738 P 5641383881414581a29d2ed15ba0fc36: Deleting consensus metadata
I20260927 22:06:51.688446 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 22746
W20260927 22:06:51.688711 22542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.159.1:35525 (error 108)
I20260927 22:06:51.692278 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.691405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35525 (local address 127.0.0.1:46492)
0927 22:06:51.691941 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.692088 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":304,"threads_started":1}
W20260927 22:06:51.692631 22542 catalog_manager.cc:4584] TS 5641383881414581a29d2ed15ba0fc36 (127.3.159.1:35525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 341795a9d1bd4a31ad56fcb2ad738738: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35525: connect: Connection refused (error 111)
I20260927 22:06:51.700218 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.689587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.3.159.2:35983)
0927 22:06:51.689961 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:51.689969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:51.690072 (+ 103us) server_negotiation.cc:407] Connection header received
0927 22:06:51.690376 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:51.690382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:51.690544 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:51.690713 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:51.691538 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.692823 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.695052 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.695798 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.695990 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.697923 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:51.697937 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:51.697963 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:51.699027 (+ 1064us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:51.699130 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:51.699167 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:51.699341 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:51.699596 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 22:06:51.699769 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260927 22:06:51.701918 22868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:51.747160 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.746398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35525 (local address 127.0.0.1:46504)
0927 22:06:51.746783 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.746966 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260927 22:06:51.748960 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.748298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35525 (local address 127.0.0.1:46514)
0927 22:06:51.748573 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.748716 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 22:06:51.805552 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 22746
I20260927 22:06:51.810437 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.809479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35525 (local address 127.0.0.1:46522)
0927 22:06:51.809847 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.810002 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 22:06:51.812216 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.811581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35525 (local address 127.0.0.1:46538)
0927 22:06:51.811826 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.811958 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:06:51.829008 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 22877
I20260927 22:06:51.839272 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.830489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.3.159.3:33255)
0927 22:06:51.830930 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:51.830935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:51.830956 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:06:51.831020 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:51.831025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:51.831154 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:51.831306 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:51.832187 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.833215 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.835031 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:51.835563 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:51.835731 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:51.837119 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:51.837129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:51.837145 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:51.838188 (+ 1043us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:51.838259 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:51.838304 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:51.838497 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:51.838683 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 22:06:51.838829 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 22:06:51.840297 22999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:51.894048 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.893197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35525 (local address 127.0.0.1:46540)
0927 22:06:51.893685 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.893837 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260927 22:06:51.895471 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:51.894899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:35525 (local address 127.0.0.1:46556)
0927 22:06:51.895185 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:51.895300 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.1:35525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157}
I20260927 22:06:51.976670 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 22877
I20260927 22:06:52.002491 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 22525
I20260927 22:06:52.004277 22586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:52.129048 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 22525
2026-09-27T22:06:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3875 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:06:52 dist-test-slave-sg4j krb5kdc[23058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:06:52 dist-test-slave-sg4j krb5kdc[23058](info): set up 2 sockets
Sep 27 22:06:52 dist-test-slave-sg4j krb5kdc[23058](info): commencing operation
krb5kdc: starting...
W20260927 22:06:53.302529 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:06:53 dist-test-slave-sg4j krb5kdc[23058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:06:53.626294 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:34801
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:34801
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:06:54.089747 23071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:54.090158 23071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:54.090343 23071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:54.099232 23071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:06:54.099411 23071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:54.099701 23071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:54.099851 23071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:06:54.099931 23071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:06:54.113265 23071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:34801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:34801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:54.115355 23071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:54.117134 23071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:54.134982 23076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:54.135066 23077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:54.135109 23079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:54.136387 23071 server_base.cc:1034] running on GCE node
I20260927 22:06:54.137668 23071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:54.140141 23071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:54.141543 23071 hybrid_clock.cc:648] HybridClock initialized: now 1790546814141363 us; error 48 us; skew 500 ppm
I20260927 22:06:54.142083 23071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:54 dist-test-slave-sg4j krb5kdc[23058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546814, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:54.147306 23071 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:06:54.152297 23071 webserver.cc:460] Webserver started at http://127.3.159.62:43487/ using document root <none> and password file <none>
I20260927 22:06:54.153941 23071 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:54.154182 23071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:54.154771 23071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:54.161497 23071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "4eae26e4323c4c3282e6ca06d66443a6"
format_stamp: "Formatted at 2026-09-27 22:06:54 on dist-test-slave-sg4j"
server_key: "6fbd454f86591da79a69398e8c21715f"
server_key_iv: "d0426242f87ef4e652b7af093e6161b4"
server_key_version: "encryptionkey@0"
I20260927 22:06:54.163259 23071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "4eae26e4323c4c3282e6ca06d66443a6"
format_stamp: "Formatted at 2026-09-27 22:06:54 on dist-test-slave-sg4j"
server_key: "6fbd454f86591da79a69398e8c21715f"
server_key_iv: "d0426242f87ef4e652b7af093e6161b4"
server_key_version: "encryptionkey@0"
I20260927 22:06:54.175122 23071 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.010s sys 0.001s
I20260927 22:06:54.184367 23086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:54.194206 23071 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.005s sys 0.002s
I20260927 22:06:54.194983 23071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "4eae26e4323c4c3282e6ca06d66443a6"
format_stamp: "Formatted at 2026-09-27 22:06:54 on dist-test-slave-sg4j"
server_key: "6fbd454f86591da79a69398e8c21715f"
server_key_iv: "d0426242f87ef4e652b7af093e6161b4"
server_key_version: "encryptionkey@0"
I20260927 22:06:54.195652 23071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:54.221581 23071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:54.228327 23071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:54.228639 23071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:54.259104 23071 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:34801
I20260927 22:06:54.259198 23137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:34801 every 8 connection(s)
I20260927 22:06:54.262008 23071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:06:54.263502 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 23071
I20260927 22:06:54.263727 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:06:54.264256 3708 external_mini_cluster.cc:1428] Setting key 45976f65ac73378db04313a4a60b5b75
I20260927 22:06:54.270808 23138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:06:54 dist-test-slave-sg4j krb5kdc[23058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:54.290167 23138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6: Bootstrap starting.
I20260927 22:06:54.296970 23138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:54.298825 23138 log.cc:824] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:54.298899 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:54.266848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.3.159.62:34801)
0927 22:06:54.267885 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:54.267906 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:54.267961 (+ 55us) server_negotiation.cc:407] Connection header received
0927 22:06:54.270265 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:54.270298 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:54.270862 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:54.271380 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:54.274118 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:54.276109 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:54.278802 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:54.279944 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:54.280177 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:54.286975 (+ 6798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:54.286989 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:54.287015 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:54.287092 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:54.291632 (+ 4540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:54.293177 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:54.293189 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:54.293198 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:54.293710 (+ 512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:54.294322 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:54.294327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:54.294329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:54.295266 (+ 937us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:54.295615 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:54.296343 (+ 728us) server_negotiation.cc:299] Negotiation successful
0927 22:06:54.297072 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":427,"threads_started":1}
I20260927 22:06:54.303722 23138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6: No bootstrap required, opened a new log
I20260927 22:06:54.312323 23138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eae26e4323c4c3282e6ca06d66443a6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34801 } }
I20260927 22:06:54.312739 23138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:54.312811 23138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eae26e4323c4c3282e6ca06d66443a6, State: Initialized, Role: FOLLOWER
I20260927 22:06:54.313505 23138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eae26e4323c4c3282e6ca06d66443a6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34801 } }
I20260927 22:06:54.313766 23138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:06:54.313920 23138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:06:54.314162 23138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:54.317806 23138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eae26e4323c4c3282e6ca06d66443a6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34801 } }
I20260927 22:06:54.318547 23138 leader_election.cc:302] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eae26e4323c4c3282e6ca06d66443a6; no voters:
I20260927 22:06:54.319360 23138 leader_election.cc:288] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:06:54.319825 23143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:06:54.321002 23143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [term 1 LEADER]: Becoming Leader. State: Replica: 4eae26e4323c4c3282e6ca06d66443a6, State: Running, Role: LEADER
I20260927 22:06:54.321835 23143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eae26e4323c4c3282e6ca06d66443a6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34801 } }
I20260927 22:06:54.323009 23138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:06:54.325675 23144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eae26e4323c4c3282e6ca06d66443a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eae26e4323c4c3282e6ca06d66443a6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34801 } } }
I20260927 22:06:54.326486 23144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:54.328210 23145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eae26e4323c4c3282e6ca06d66443a6. Latest consensus state: current_term: 1 leader_uuid: "4eae26e4323c4c3282e6ca06d66443a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eae26e4323c4c3282e6ca06d66443a6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 34801 } } }
I20260927 22:06:54.328799 23145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:54.330322 23152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:06:54.336267 23152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:06:54.346503 23152 catalog_manager.cc:1347] Generated new cluster ID: 6d7f43aa08a847d4bc6cc15246de7e55
I20260927 22:06:54.346658 23152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:06:54.377894 23152 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:06:54.380525 23152 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:06:54.399262 23152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6: Generated new TSK 0
I20260927 22:06:54.400630 23152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.159.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
Entry for principal kudu/127.3.159.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab.
I20260927 22:06:54.500600 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34801
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:06:54.964120 23164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:54.964547 23164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:54.964663 23164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:54.973753 23164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:54.973915 23164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:54.974135 23164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:06:54.985852 23164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.1.keytab
--principal=kudu/127.3.159.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:54.987637 23164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:54.989341 23164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:55.006987 23169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:55.007025 23170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:55.007243 23172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:55.009579 23164 server_base.cc:1034] running on GCE node
I20260927 22:06:55.010495 23164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:55.012188 23164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:55.013518 23164 hybrid_clock.cc:648] HybridClock initialized: now 1790546815013428 us; error 95 us; skew 500 ppm
I20260927 22:06:55.014025 23164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:55 dist-test-slave-sg4j krb5kdc[23058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:55.018857 23164 init.cc:377] Logged in from keytab as kudu/127.3.159.1@KRBTEST.COM (short username kudu)
I20260927 22:06:55.021173 23164 webserver.cc:460] Webserver started at http://127.3.159.1:45553/ using document root <none> and password file <none>
I20260927 22:06:55.022233 23164 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:55.022374 23164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:55.023118 23164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:55.027954 23164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "9a0482928bd14d84aa72bc4223676903"
format_stamp: "Formatted at 2026-09-27 22:06:55 on dist-test-slave-sg4j"
server_key: "fcd1694522a3ba22ad76797628965dba"
server_key_iv: "43607c7fd02ae53aa527a5b877d29590"
server_key_version: "encryptionkey@0"
I20260927 22:06:55.029121 23164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "9a0482928bd14d84aa72bc4223676903"
format_stamp: "Formatted at 2026-09-27 22:06:55 on dist-test-slave-sg4j"
server_key: "fcd1694522a3ba22ad76797628965dba"
server_key_iv: "43607c7fd02ae53aa527a5b877d29590"
server_key_version: "encryptionkey@0"
I20260927 22:06:55.036947 23164 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260927 22:06:55.042721 23179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:55.045001 23164 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:06:55.045336 23164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "9a0482928bd14d84aa72bc4223676903"
format_stamp: "Formatted at 2026-09-27 22:06:55 on dist-test-slave-sg4j"
server_key: "fcd1694522a3ba22ad76797628965dba"
server_key_iv: "43607c7fd02ae53aa527a5b877d29590"
server_key_version: "encryptionkey@0"
I20260927 22:06:55.045750 23164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:55.073000 23164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:55.080034 23164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:55.080477 23164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:55.082041 23164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:55.083998 23164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:55.084123 23164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:55.084302 23164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:55.084390 23164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:55.130607 23164 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:45103
I20260927 22:06:55.130626 23291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:45103 every 8 connection(s)
I20260927 22:06:55.132335 23164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:06:55.134099 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 23164
I20260927 22:06:55.134310 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:06:55.134909 3708 external_mini_cluster.cc:1428] Setting key d6fb436f08899008875c535c02bc7790
Sep 27 22:06:55 dist-test-slave-sg4j krb5kdc[23058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.1@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:55.163812 23294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:55.135811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:52067 (local address 127.3.159.62:34801)
0927 22:06:55.136492 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:55.136500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:55.137677 (+ 1177us) server_negotiation.cc:407] Connection header received
0927 22:06:55.139904 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:55.139913 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:55.140049 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:55.140285 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:55.142522 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:55.143794 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:55.146586 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:55.147411 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:55.147625 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:55.153347 (+ 5722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:55.153359 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:55.153367 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:55.153480 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:55.157947 (+ 4467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:55.159538 (+ 1591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:55.159548 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:55.159555 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:55.159781 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:55.160486 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:55.160495 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:55.160501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:55.161029 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:55.161242 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:55.162834 (+ 1592us) server_negotiation.cc:299] Negotiation successful
0927 22:06:55.163237 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":324,"threads_started":1}
I20260927 22:06:55.165004 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:55.136467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:34801 (local address 127.3.159.1:52067)
0927 22:06:55.137468 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:55.137533 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:55.139510 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:55.140530 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:55.140539 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:55.141492 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:55.142240 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:55.142264 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:55.144654 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:55.144661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:55.146301 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:55.146325 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:55.147869 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:55.147877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:55.148223 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:55.149373 (+ 1150us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:06:55.149641 (+ 268us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:55.152867 (+ 3226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:55.158269 (+ 5402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:55.158281 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:55.158311 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:55.159283 (+ 972us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:55.159999 (+ 716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:55.160004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:55.160006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:55.160312 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:55.161292 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:55.161303 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:55.162120 (+ 817us) client_negotiation.cc:769] Sending connection context
0927 22:06:55.162665 (+ 545us) client_negotiation.cc:240] Negotiation successful
0927 22:06:55.163400 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":404,"threads_started":1}
I20260927 22:06:55.168023 23292 heartbeater.cc:347] Connected to a master server at 127.3.159.62:34801
I20260927 22:06:55.168577 23292 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:55.169644 23292 heartbeater.cc:511] Master 127.3.159.62:34801 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.159.2@KRBTEST.COM; defaulting to no policy
I20260927 22:06:55.173125 23103 ts_manager.cc:194] Registered new tserver with Master: 9a0482928bd14d84aa72bc4223676903 (127.3.159.1:45103)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.2@KRBTEST.COM" created.
I20260927 22:06:55.175451 23103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.1@KRBTEST.COM'} at 127.3.159.1:52067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
Entry for principal kudu/127.3.159.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab.
I20260927 22:06:55.215252 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34801
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:06:55.690708 23298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:55.691150 23298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:55.691265 23298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:55.699936 23298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:55.700115 23298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:55.700249 23298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:06:55.712955 23298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.2.keytab
--principal=kudu/127.3.159.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:55.714996 23298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:55.716930 23298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:55.736539 23306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:55.736613 23304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:55.736683 23303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:55.737810 23298 server_base.cc:1034] running on GCE node
I20260927 22:06:55.738914 23298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:55.740854 23298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:55.742195 23298 hybrid_clock.cc:648] HybridClock initialized: now 1790546815742123 us; error 69 us; skew 500 ppm
I20260927 22:06:55.742740 23298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:55 dist-test-slave-sg4j krb5kdc[23058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:55.755657 23298 init.cc:377] Logged in from keytab as kudu/127.3.159.2@KRBTEST.COM (short username kudu)
I20260927 22:06:55.758559 23298 webserver.cc:460] Webserver started at http://127.3.159.2:38331/ using document root <none> and password file <none>
I20260927 22:06:55.759727 23298 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:55.759956 23298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:55.760380 23298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:55.764319 23298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f"
format_stamp: "Formatted at 2026-09-27 22:06:55 on dist-test-slave-sg4j"
server_key: "2d77a7d91fc8dd9c32c4ace5ce786289"
server_key_iv: "f5997baed16854a88b65c260c52c25d2"
server_key_version: "encryptionkey@0"
I20260927 22:06:55.765255 23298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f"
format_stamp: "Formatted at 2026-09-27 22:06:55 on dist-test-slave-sg4j"
server_key: "2d77a7d91fc8dd9c32c4ace5ce786289"
server_key_iv: "f5997baed16854a88b65c260c52c25d2"
server_key_version: "encryptionkey@0"
I20260927 22:06:55.772696 23298 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 22:06:55.778155 23313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:55.780752 23298 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:06:55.781049 23298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f"
format_stamp: "Formatted at 2026-09-27 22:06:55 on dist-test-slave-sg4j"
server_key: "2d77a7d91fc8dd9c32c4ace5ce786289"
server_key_iv: "f5997baed16854a88b65c260c52c25d2"
server_key_version: "encryptionkey@0"
I20260927 22:06:55.781344 23298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:55.803947 23298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:55.811201 23298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:55.811630 23298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:55.813082 23298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:55.815124 23298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:55.815244 23298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:55.815419 23298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:55.815502 23298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:55.862823 23298 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:43225
I20260927 22:06:55.862952 23425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:43225 every 8 connection(s)
I20260927 22:06:55.864692 23298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:06:55.873271 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 23298
I20260927 22:06:55.873579 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:06:55.874114 3708 external_mini_cluster.cc:1428] Setting key 075d8df335e2f7b618ee86cfe45248a3
Sep 27 22:06:55 dist-test-slave-sg4j krb5kdc[23058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.2@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:55.889772 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:55.868596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:46083 (local address 127.3.159.62:34801)
0927 22:06:55.869153 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:55.869158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:55.870788 (+ 1630us) server_negotiation.cc:407] Connection header received
0927 22:06:55.873663 (+ 2875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:55.873669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:55.873799 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:55.873940 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:55.875970 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:55.876748 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:55.878537 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:55.878991 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:55.879151 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:55.883095 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:55.883105 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:55.883108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:55.883147 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:55.886088 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:55.886943 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:55.886956 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:55.886959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:55.887194 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:55.887682 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:55.887694 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:55.887698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:55.888026 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:55.888220 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:55.889049 (+ 829us) server_negotiation.cc:299] Negotiation successful
0927 22:06:55.889339 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":321,"threads_started":1}
I20260927 22:06:55.890894 23429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:55.869339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:34801 (local address 127.3.159.2:46083)
0927 22:06:55.870501 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:55.870564 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:55.873116 (+ 2552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:55.874257 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:55.874268 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:55.875054 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:55.875746 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:55.875767 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:55.876941 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:55.876945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:55.878364 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:55.878378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:55.879267 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:55.879275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:55.879533 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:55.880314 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:06:55.880344 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:55.882788 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:55.886269 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:55.886279 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:55.886308 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:55.886796 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:55.887322 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:55.887327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:55.887331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:55.887548 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:55.888162 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:55.888171 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:55.888632 (+ 461us) client_negotiation.cc:769] Sending connection context
0927 22:06:55.888942 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 22:06:55.889578 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":468,"threads_started":1}
I20260927 22:06:55.893075 23426 heartbeater.cc:347] Connected to a master server at 127.3.159.62:34801
I20260927 22:06:55.893554 23426 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:55.894582 23426 heartbeater.cc:511] Master 127.3.159.62:34801 requested a full tablet report, sending...
I20260927 22:06:55.896844 23103 ts_manager.cc:194] Registered new tserver with Master: cb8ed0ec1f3a400aafcd3817c68c0d6f (127.3.159.2:43225)
I20260927 22:06:55.898043 23103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.2@KRBTEST.COM'} at 127.3.159.2:46083
WARNING: no policy specified for kudu/127.3.159.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
Entry for principal kudu/127.3.159.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab.
I20260927 22:06:55.953579 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34801
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:06:56.180750 23292 heartbeater.cc:503] Master 127.3.159.62:34801 was elected leader, sending a full tablet report...
W20260927 22:06:56.364307 23432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:56.365006 23432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:56.365202 23432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:56.375352 23432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:56.375598 23432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:56.375823 23432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:06:56.388702 23432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.3.keytab
--principal=kudu/127.3.159.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:34801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:56.390970 23432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:56.393070 23432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:56.413657 23440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:56.413507 23438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:56.413494 23437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:56.420604 23432 server_base.cc:1034] running on GCE node
I20260927 22:06:56.421818 23432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:56.423436 23432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:56.424943 23432 hybrid_clock.cc:648] HybridClock initialized: now 1790546816424856 us; error 80 us; skew 500 ppm
I20260927 22:06:56.425810 23432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:06:56 dist-test-slave-sg4j krb5kdc[23058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546816, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:06:56.431517 23432 init.cc:377] Logged in from keytab as kudu/127.3.159.3@KRBTEST.COM (short username kudu)
I20260927 22:06:56.434378 23432 webserver.cc:460] Webserver started at http://127.3.159.3:39441/ using document root <none> and password file <none>
I20260927 22:06:56.435495 23432 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:56.435699 23432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:56.436120 23432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:56.440337 23432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "ed77db8a2fce468dad3804e25194dd32"
format_stamp: "Formatted at 2026-09-27 22:06:56 on dist-test-slave-sg4j"
server_key: "a88b8d6976715734b54846d6309c6b32"
server_key_iv: "019422f213defe65ef257d59bbfa90b2"
server_key_version: "encryptionkey@0"
I20260927 22:06:56.441583 23432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "ed77db8a2fce468dad3804e25194dd32"
format_stamp: "Formatted at 2026-09-27 22:06:56 on dist-test-slave-sg4j"
server_key: "a88b8d6976715734b54846d6309c6b32"
server_key_iv: "019422f213defe65ef257d59bbfa90b2"
server_key_version: "encryptionkey@0"
I20260927 22:06:56.450124 23432 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260927 22:06:56.455518 23447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:56.457834 23432 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:06:56.458110 23432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "ed77db8a2fce468dad3804e25194dd32"
format_stamp: "Formatted at 2026-09-27 22:06:56 on dist-test-slave-sg4j"
server_key: "a88b8d6976715734b54846d6309c6b32"
server_key_iv: "019422f213defe65ef257d59bbfa90b2"
server_key_version: "encryptionkey@0"
I20260927 22:06:56.458397 23432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:56.490520 23432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:56.496281 23432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:56.496661 23432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:56.498004 23432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:56.499778 23432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:56.499897 23432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:56.500072 23432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:56.500159 23432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:56.546412 23432 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:40479
I20260927 22:06:56.546480 23559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:40479 every 8 connection(s)
I20260927 22:06:56.548282 23432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:06:56.550055 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 23432
I20260927 22:06:56.550309 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:06:56.551051 3708 external_mini_cluster.cc:1428] Setting key 82a1a7435c5b7d1e9f626cfc1ab64118
Sep 27 22:06:56 dist-test-slave-sg4j krb5kdc[23058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546816, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.3@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:06:56.576926 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.552448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:44997 (local address 127.3.159.62:34801)
0927 22:06:56.553033 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:56.553038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:56.554707 (+ 1669us) server_negotiation.cc:407] Connection header received
0927 22:06:56.557283 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:56.557289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:56.557434 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:56.557615 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:56.559568 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.560424 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.562655 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.563141 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.563302 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.567905 (+ 4603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:56.567912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:56.567917 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:56.567974 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:56.571680 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:56.572779 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:56.572784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:56.572787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:56.572968 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:56.574107 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:56.574112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:56.574115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:56.574562 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:56.574734 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:56.576088 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0927 22:06:56.576411 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":298,"threads_started":1}
I20260927 22:06:56.577837 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.553282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:34801 (local address 127.3.159.3:44997)
0927 22:06:56.554489 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:56.554544 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:56.556910 (+ 2366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:56.557946 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:56.557955 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:56.558627 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:56.559342 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.559364 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.560622 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.560628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.562468 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.562490 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.563442 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.563451 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.563786 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.564741 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:06:56.564769 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:56.567600 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:56.571923 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:56.571933 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:56.571950 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:56.572582 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:56.573186 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:06:56.573191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:06:56.573195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:06:56.573859 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:06:56.574845 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:56.574857 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:56.575503 (+ 646us) client_negotiation.cc:769] Sending connection context
0927 22:06:56.575890 (+ 387us) client_negotiation.cc:240] Negotiation successful
0927 22:06:56.576390 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":470,"threads_started":1}
I20260927 22:06:56.580375 23560 heartbeater.cc:347] Connected to a master server at 127.3.159.62:34801
I20260927 22:06:56.580803 23560 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:56.581864 23560 heartbeater.cc:511] Master 127.3.159.62:34801 requested a full tablet report, sending...
I20260927 22:06:56.584584 23103 ts_manager.cc:194] Registered new tserver with Master: ed77db8a2fce468dad3804e25194dd32 (127.3.159.3:40479)
I20260927 22:06:56.585773 23103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.159.3@KRBTEST.COM'} at 127.3.159.3:44997
I20260927 22:06:56.597849 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:06:56.623881 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.603863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.3.159.62:34801)
0927 22:06:56.604177 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:56.604185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:56.604734 (+ 549us) server_negotiation.cc:407] Connection header received
0927 22:06:56.604942 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:56.604947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:56.605055 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:56.605180 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:56.607307 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.608598 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.610739 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.611243 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.611403 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.615137 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:56.615144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:56.615148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:56.615205 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:56.618629 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:56.620115 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:56.620122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:56.620125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:56.620364 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:56.621182 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:56.621189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:56.621193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:56.621845 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:56.622115 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:56.622995 (+ 880us) server_negotiation.cc:299] Negotiation successful
0927 22:06:56.623352 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 22:06:56.631022 23103 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:06:56.641072 23103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:06:56.689679 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.660078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.3.159.3:40479)
0927 22:06:56.675235 (+ 15157us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:56.675252 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:56.676719 (+ 1467us) server_negotiation.cc:407] Connection header received
0927 22:06:56.677251 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:56.677260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:56.677602 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:56.677843 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:56.679122 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.681078 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.684953 (+ 3875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.687507 (+ 2554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.688082 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.688438 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:56.688889 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 22:06:56.689144 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14853,"thread_start_us":15176,"threads_started":1}
I20260927 22:06:56.689994 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.659386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:40479 (local address 127.0.0.1:38972)
0927 22:06:56.676403 (+ 17017us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:56.676460 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:56.676892 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:56.678173 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:56.678183 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:56.678228 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:56.678796 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.678818 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.681269 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.681281 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.684437 (+ 3156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.684461 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.688132 (+ 3671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.688140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.688478 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.688563 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 22:06:56.688761 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 22:06:56.689145 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16722,"thread_start_us":19566,"threads_started":1}
I20260927 22:06:56.691505 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.665834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:43225 (local address 127.0.0.1:57790)
0927 22:06:56.679306 (+ 13472us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:56.679336 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:56.679482 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:56.680487 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:56.680493 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:56.680531 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:56.681080 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.681097 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.683292 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.683300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.686034 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.686060 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.690307 (+ 4247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.690317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.690555 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.690611 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 22:06:56.690745 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 22:06:56.690973 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13290,"mutex_wait_us":6,"thread_start_us":10060,"threads_started":1}
I20260927 22:06:56.691586 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.666186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.3.159.2:43225)
0927 22:06:56.678701 (+ 12515us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:56.678714 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:56.679489 (+ 775us) server_negotiation.cc:407] Connection header received
0927 22:06:56.679694 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:56.679709 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:56.680193 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:56.680509 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:56.681563 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.683112 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.686332 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.689621 (+ 3289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.690364 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.690583 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:56.690956 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 22:06:56.691188 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12282,"thread_start_us":12135,"threads_started":1}
I20260927 22:06:56.692297 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.659428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.3.159.1:45103)
0927 22:06:56.680356 (+ 20928us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:56.680364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:56.680403 (+ 39us) server_negotiation.cc:407] Connection header received
0927 22:06:56.680524 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:56.680533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:56.680827 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:56.682085 (+ 1258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:56.682350 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.684275 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.687786 (+ 3511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.690004 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.690159 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.690501 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:56.691673 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0927 22:06:56.691897 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20670,"thread_start_us":20367,"threads_started":1}
I20260927 22:06:56.692945 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.659021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:45103 (local address 127.0.0.1:52964)
0927 22:06:56.676505 (+ 17484us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:56.676549 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:56.676878 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:56.681132 (+ 4254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:56.681141 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:56.681182 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:56.681791 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.681815 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.684510 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.684517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.687338 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.687362 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.690746 (+ 3384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.690755 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.691036 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.691082 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 22:06:56.691209 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 22:06:56.691434 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16441,"mutex_wait_us":40,"thread_start_us":16668,"threads_started":1}
I20260927 22:06:56.695802 23495 tablet_service.cc:1467] Processing CreateTablet for tablet c864b1dddfa546da8151bfc73df39ba6 (DEFAULT_TABLE table=test-table [id=53d4bfbab4294e759f492bd5595d094b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:56.697959 23495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c864b1dddfa546da8151bfc73df39ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:56.697018 23361 tablet_service.cc:1467] Processing CreateTablet for tablet c864b1dddfa546da8151bfc73df39ba6 (DEFAULT_TABLE table=test-table [id=53d4bfbab4294e759f492bd5595d094b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:56.699298 23361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c864b1dddfa546da8151bfc73df39ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:56.701833 23227 tablet_service.cc:1467] Processing CreateTablet for tablet c864b1dddfa546da8151bfc73df39ba6 (DEFAULT_TABLE table=test-table [id=53d4bfbab4294e759f492bd5595d094b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:06:56.704169 23227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c864b1dddfa546da8151bfc73df39ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:56.714159 23579 tablet_bootstrap.cc:492] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: Bootstrap starting.
I20260927 22:06:56.716727 23580 tablet_bootstrap.cc:492] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: Bootstrap starting.
I20260927 22:06:56.719013 23579 tablet_bootstrap.cc:654] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:56.720496 23579 log.cc:824] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:56.721985 23580 tablet_bootstrap.cc:654] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:56.723294 23582 tablet_bootstrap.cc:492] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: Bootstrap starting.
I20260927 22:06:56.723642 23580 log.cc:824] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:56.724917 23579 tablet_bootstrap.cc:492] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: No bootstrap required, opened a new log
I20260927 22:06:56.725258 23579 ts_tablet_manager.cc:1397] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260927 22:06:56.728747 23580 tablet_bootstrap.cc:492] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: No bootstrap required, opened a new log
I20260927 22:06:56.728945 23582 tablet_bootstrap.cc:654] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:56.729652 23580 ts_tablet_manager.cc:1397] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260927 22:06:56.730935 23582 log.cc:824] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:56.735261 23582 tablet_bootstrap.cc:492] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: No bootstrap required, opened a new log
I20260927 22:06:56.735693 23582 ts_tablet_manager.cc:1397] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260927 22:06:56.738626 23579 raft_consensus.cc:348] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0482928bd14d84aa72bc4223676903" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45103 } } peers { permanent_uuid: "ed77db8a2fce468dad3804e25194dd32" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40479 } } peers { permanent_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43225 } }
I20260927 22:06:56.739410 23579 raft_consensus.cc:374] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:56.739594 23579 raft_consensus.cc:729] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8ed0ec1f3a400aafcd3817c68c0d6f, State: Initialized, Role: FOLLOWER
I20260927 22:06:56.740518 23579 consensus_queue.cc:260] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0482928bd14d84aa72bc4223676903" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45103 } } peers { permanent_uuid: "ed77db8a2fce468dad3804e25194dd32" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40479 } } peers { permanent_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43225 } }
I20260927 22:06:56.742159 23426 heartbeater.cc:503] Master 127.3.159.62:34801 was elected leader, sending a full tablet report...
I20260927 22:06:56.743119 23579 ts_tablet_manager.cc:1428] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260927 22:06:56.745492 23580 raft_consensus.cc:348] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0482928bd14d84aa72bc4223676903" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45103 } } peers { permanent_uuid: "ed77db8a2fce468dad3804e25194dd32" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40479 } } peers { permanent_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43225 } }
I20260927 22:06:56.746037 23580 raft_consensus.cc:374] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:56.746237 23580 raft_consensus.cc:729] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed77db8a2fce468dad3804e25194dd32, State: Initialized, Role: FOLLOWER
I20260927 22:06:56.747012 23580 consensus_queue.cc:260] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0482928bd14d84aa72bc4223676903" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45103 } } peers { permanent_uuid: "ed77db8a2fce468dad3804e25194dd32" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40479 } } peers { permanent_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43225 } }
I20260927 22:06:56.748288 23582 raft_consensus.cc:348] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0482928bd14d84aa72bc4223676903" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45103 } } peers { permanent_uuid: "ed77db8a2fce468dad3804e25194dd32" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40479 } } peers { permanent_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43225 } }
I20260927 22:06:56.748816 23582 raft_consensus.cc:374] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:56.748945 23582 raft_consensus.cc:729] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a0482928bd14d84aa72bc4223676903, State: Initialized, Role: FOLLOWER
I20260927 22:06:56.749169 23580 ts_tablet_manager.cc:1428] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: Time spent starting tablet: real 0.019s user 0.016s sys 0.000s
I20260927 22:06:56.749562 23560 heartbeater.cc:503] Master 127.3.159.62:34801 was elected leader, sending a full tablet report...
I20260927 22:06:56.749826 23582 consensus_queue.cc:260] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0482928bd14d84aa72bc4223676903" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45103 } } peers { permanent_uuid: "ed77db8a2fce468dad3804e25194dd32" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40479 } } peers { permanent_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43225 } }
I20260927 22:06:56.752457 23582 ts_tablet_manager.cc:1428] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
I20260927 22:06:56.779254 23586 raft_consensus.cc:482] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:06:56.779637 23586 raft_consensus.cc:504] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0482928bd14d84aa72bc4223676903" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45103 } } peers { permanent_uuid: "ed77db8a2fce468dad3804e25194dd32" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40479 } } peers { permanent_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43225 } }
I20260927 22:06:56.782280 23586 leader_election.cc:288] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a0482928bd14d84aa72bc4223676903 (127.3.159.1:45103), cb8ed0ec1f3a400aafcd3817c68c0d6f (127.3.159.2:43225)
I20260927 22:06:56.792205 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.783251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:47507 (local address 127.3.159.1:45103)
0927 22:06:56.783558 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:56.783565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:56.783596 (+ 31us) server_negotiation.cc:407] Connection header received
0927 22:06:56.783722 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:56.783728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:56.783832 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:56.784005 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:56.784679 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.786107 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.788470 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.790086 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.790274 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.790396 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:56.791423 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0927 22:06:56.791892 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 22:06:56.792201 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.782647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:45103 (local address 127.3.159.3:47507)
0927 22:06:56.783067 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:56.783095 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:56.783284 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:56.784115 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:56.784123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:56.784171 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:56.784418 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.784441 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.786298 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.786305 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.788226 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.788246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.790387 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.790396 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.790970 (+ 574us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.791151 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 22:06:56.791323 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 22:06:56.791801 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260927 22:06:56.792928 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.782792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:43225 (local address 127.3.159.3:58383)
0927 22:06:56.783936 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:56.783963 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:56.784126 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:56.784674 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:56.784680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:56.784715 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:06:56.785227 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.785244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.787142 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.787148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.789305 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:56.789332 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.791822 (+ 2490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.791830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:56.792028 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.792111 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 22:06:56.792266 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 22:06:56.792492 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"thread_start_us":593,"threads_started":1}
I20260927 22:06:56.793485 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.783097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:58383 (local address 127.3.159.2:43225)
0927 22:06:56.783669 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:56.783680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:56.784092 (+ 412us) server_negotiation.cc:407] Connection header received
0927 22:06:56.784319 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:56.784325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:56.784416 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:56.784550 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:06:56.785563 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.786757 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.789685 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.791310 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.791729 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.791886 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:56.792572 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 22:06:56.792936 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260927 22:06:56.794363 23247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c864b1dddfa546da8151bfc73df39ba6" candidate_uuid: "ed77db8a2fce468dad3804e25194dd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a0482928bd14d84aa72bc4223676903" is_pre_election: true
I20260927 22:06:56.794483 23381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c864b1dddfa546da8151bfc73df39ba6" candidate_uuid: "ed77db8a2fce468dad3804e25194dd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" is_pre_election: true
I20260927 22:06:56.795101 23247 raft_consensus.cc:2454] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed77db8a2fce468dad3804e25194dd32 in term 0.
I20260927 22:06:56.795113 23381 raft_consensus.cc:2454] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed77db8a2fce468dad3804e25194dd32 in term 0.
I20260927 22:06:56.795996 23448 leader_election.cc:302] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb8ed0ec1f3a400aafcd3817c68c0d6f, ed77db8a2fce468dad3804e25194dd32; no voters:
I20260927 22:06:56.796586 23586 raft_consensus.cc:2781] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:06:56.796832 23586 raft_consensus.cc:482] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:06:56.796957 23586 raft_consensus.cc:3037] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:56.799916 23586 raft_consensus.cc:504] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0482928bd14d84aa72bc4223676903" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45103 } } peers { permanent_uuid: "ed77db8a2fce468dad3804e25194dd32" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40479 } } peers { permanent_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43225 } }
W20260927 22:06:56.800662 23561 tablet.cc:2369] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:06:56.801072 23586 leader_election.cc:288] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [CANDIDATE]: Term 1 election: Requested vote from peers 9a0482928bd14d84aa72bc4223676903 (127.3.159.1:45103), cb8ed0ec1f3a400aafcd3817c68c0d6f (127.3.159.2:43225)
I20260927 22:06:56.801968 23381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c864b1dddfa546da8151bfc73df39ba6" candidate_uuid: "ed77db8a2fce468dad3804e25194dd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f"
I20260927 22:06:56.801847 23247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c864b1dddfa546da8151bfc73df39ba6" candidate_uuid: "ed77db8a2fce468dad3804e25194dd32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a0482928bd14d84aa72bc4223676903"
I20260927 22:06:56.802280 23381 raft_consensus.cc:3037] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:56.802338 23247 raft_consensus.cc:3037] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:56.806558 23247 raft_consensus.cc:2454] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed77db8a2fce468dad3804e25194dd32 in term 1.
I20260927 22:06:56.806980 23381 raft_consensus.cc:2454] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed77db8a2fce468dad3804e25194dd32 in term 1.
I20260927 22:06:56.807319 23450 leader_election.cc:302] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a0482928bd14d84aa72bc4223676903, ed77db8a2fce468dad3804e25194dd32; no voters:
I20260927 22:06:56.808350 23586 raft_consensus.cc:2781] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:06:56.809059 23586 raft_consensus.cc:686] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 1 LEADER]: Becoming Leader. State: Replica: ed77db8a2fce468dad3804e25194dd32, State: Running, Role: LEADER
I20260927 22:06:56.809769 23586 consensus_queue.cc:237] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0482928bd14d84aa72bc4223676903" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45103 } } peers { permanent_uuid: "ed77db8a2fce468dad3804e25194dd32" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40479 } } peers { permanent_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43225 } }
I20260927 22:06:56.816310 23103 catalog_manager.cc:5495] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed77db8a2fce468dad3804e25194dd32 (127.3.159.3). New cstate: current_term: 1 leader_uuid: "ed77db8a2fce468dad3804e25194dd32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a0482928bd14d84aa72bc4223676903" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed77db8a2fce468dad3804e25194dd32" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 40479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43225 } health_report { overall_health: UNKNOWN } } }
I20260927 22:06:56.855683 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.848269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.3.159.3:40479)
0927 22:06:56.848619 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:56.848626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:56.848754 (+ 128us) server_negotiation.cc:407] Connection header received
0927 22:06:56.849075 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:56.849080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:56.849196 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:56.849366 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:06:56.850047 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.850975 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.852790 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.853271 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.853546 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.854231 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:06:56.854824 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:06:56.854980 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:56.855218 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 22:06:56.855315 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 22:06:56.867630 23247 raft_consensus.cc:1261] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903 [term 1 FOLLOWER]: Refusing update from remote peer ed77db8a2fce468dad3804e25194dd32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:06:56.867825 23381 raft_consensus.cc:1261] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f [term 1 FOLLOWER]: Refusing update from remote peer ed77db8a2fce468dad3804e25194dd32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260927 22:06:56.868037 23427 tablet.cc:2369] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:06:56.869438 23586 consensus_queue.cc:1035] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a0482928bd14d84aa72bc4223676903" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 45103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:06:56.869903 23589 consensus_queue.cc:1035] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8ed0ec1f3a400aafcd3817c68c0d6f" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 43225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:06:56.884729 23592 mvcc.cc:205] Tried to move back new op lower bound from 7334079761874104320 to 7334079761658073088. Current Snapshot: MvccSnapshot[applied={T|T < 7334079761874104320}]
W20260927 22:06:56.886193 23293 tablet.cc:2369] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:06:56.886746 23593 mvcc.cc:205] Tried to move back new op lower bound from 7334079761874104320 to 7334079761658073088. Current Snapshot: MvccSnapshot[applied={T|T < 7334079761874104320}]
I20260927 22:06:56.900020 23594 mvcc.cc:205] Tried to move back new op lower bound from 7334079761874104320 to 7334079761658073088. Current Snapshot: MvccSnapshot[applied={T|T < 7334079761874104320}]
I20260927 22:06:56.915369 23103 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:06:56.915781 23103 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:06:56.920641 23103 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4eae26e4323c4c3282e6ca06d66443a6: Sending DeleteTablet for 3 replicas of tablet c864b1dddfa546da8151bfc73df39ba6
I20260927 22:06:56.922267 23495 tablet_service.cc:1514] Processing DeleteTablet for tablet c864b1dddfa546da8151bfc73df39ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:56 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:38972
I20260927 22:06:56.922250 23227 tablet_service.cc:1514] Processing DeleteTablet for tablet c864b1dddfa546da8151bfc73df39ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:56 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:52964
I20260927 22:06:56.923187 23361 tablet_service.cc:1514] Processing DeleteTablet for tablet c864b1dddfa546da8151bfc73df39ba6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:06:56 UTC) from {username='kudu', principal='kudu/127.3.159.62@KRBTEST.COM'} at 127.0.0.1:57790
I20260927 22:06:56.924341 23604 tablet_replica.cc:330] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: stopping tablet replica
I20260927 22:06:56.925091 23604 raft_consensus.cc:2229] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:06:56.925918 23604 raft_consensus.cc:2258] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:56.928651 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 23164
I20260927 22:06:56.929041 23604 ts_tablet_manager.cc:1905] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:56.938283 23604 ts_tablet_manager.cc:1918] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:56.938723 23604 log.cc:1196] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/c864b1dddfa546da8151bfc73df39ba6
I20260927 22:06:56.939531 23604 ts_tablet_manager.cc:1939] T c864b1dddfa546da8151bfc73df39ba6 P cb8ed0ec1f3a400aafcd3817c68c0d6f: Deleting consensus metadata
I20260927 22:06:56.940665 23603 tablet_replica.cc:330] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: stopping tablet replica
I20260927 22:06:56.941835 23603 raft_consensus.cc:2229] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:06:56.942394 23087 catalog_manager.cc:4841] TS cb8ed0ec1f3a400aafcd3817c68c0d6f (127.3.159.2:43225): tablet c864b1dddfa546da8151bfc73df39ba6 (table test-table [id=53d4bfbab4294e759f492bd5595d094b]) successfully deleted
I20260927 22:06:56.943631 23603 raft_consensus.cc:2258] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:56.945036 23602 tablet_replica.cc:330] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: stopping tablet replica
I20260927 22:06:56.946929 23603 ts_tablet_manager.cc:1905] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:56.946883 23602 raft_consensus.cc:2229] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:06:56.947482 23602 raft_consensus.cc:2258] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:06:56.949745 23602 ts_tablet_manager.cc:1905] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:06:56.956810 23603 ts_tablet_manager.cc:1918] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:56.957163 23603 log.cc:1196] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/c864b1dddfa546da8151bfc73df39ba6
I20260927 22:06:56.957881 23603 ts_tablet_manager.cc:1939] T c864b1dddfa546da8151bfc73df39ba6 P ed77db8a2fce468dad3804e25194dd32: Deleting consensus metadata
I20260927 22:06:56.958775 23602 ts_tablet_manager.cc:1918] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:06:56.959036 23602 log.cc:1196] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/c864b1dddfa546da8151bfc73df39ba6
Sep 27 22:06:56 dist-test-slave-sg4j krb5kdc[23058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.1@KRBTEST.COM
I20260927 22:06:56.959721 23602 ts_tablet_manager.cc:1939] T c864b1dddfa546da8151bfc73df39ba6 P 9a0482928bd14d84aa72bc4223676903: Deleting consensus metadata
I20260927 22:06:56.960280 23088 catalog_manager.cc:4841] TS ed77db8a2fce468dad3804e25194dd32 (127.3.159.3:40479): tablet c864b1dddfa546da8151bfc73df39ba6 (table test-table [id=53d4bfbab4294e759f492bd5595d094b]) successfully deleted
I20260927 22:06:56.961750 23089 catalog_manager.cc:4841] TS 9a0482928bd14d84aa72bc4223676903 (127.3.159.1:45103): tablet c864b1dddfa546da8151bfc73df39ba6 (table test-table [id=53d4bfbab4294e759f492bd5595d094b]) successfully deleted
I20260927 22:06:56.969542 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:56.930332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.3.159.1:45103)
0927 22:06:56.930580 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:56.930591 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:56.945907 (+ 15316us) server_negotiation.cc:407] Connection header received
0927 22:06:56.946516 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:56.946524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:56.946666 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:56.946834 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:56.949226 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.950703 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.952990 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:56.953832 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:56.953952 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:56.961432 (+ 7480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:56.961446 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:56.961467 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:56.961523 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:56.965529 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:56.966854 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:56.966863 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:56.966865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:56.967049 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:56.967628 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:56.967632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:56.967635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:56.968091 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:56.968344 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:56.968906 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 22:06:56.969132 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 22:06:56.970310 23287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:57.087266 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 23164
I20260927 22:06:57.115742 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 23298
Sep 27 22:06:57 dist-test-slave-sg4j krb5kdc[23058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.2@KRBTEST.COM
I20260927 22:06:57.138592 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:57.117057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.3.159.2:43225)
0927 22:06:57.117286 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:57.117295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:57.117325 (+ 30us) server_negotiation.cc:407] Connection header received
0927 22:06:57.117455 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:57.117462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:57.117588 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:57.117791 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:57.119955 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:57.121270 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:57.123316 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:57.124066 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:57.124329 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:57.130022 (+ 5693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:57.130053 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:57.130081 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:57.130152 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:57.134388 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:57.135616 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:57.135624 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:57.135627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:57.135832 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:57.136481 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:57.136494 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:57.136498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:57.136982 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:57.137331 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:57.137771 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 22:06:57.138207 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 22:06:57.139700 23421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:57.242959 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 23298
I20260927 22:06:57.265204 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 23432
Sep 27 22:06:57 dist-test-slave-sg4j krb5kdc[23058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.3@KRBTEST.COM
I20260927 22:06:57.284561 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:57.266194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.3.159.3:40479)
0927 22:06:57.266389 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:57.266395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:57.266623 (+ 228us) server_negotiation.cc:407] Connection header received
0927 22:06:57.266842 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:57.266848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:57.266932 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:57.267061 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:57.268606 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:57.270006 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:57.271424 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:57.271934 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:57.272089 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:57.277267 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:57.277286 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:57.277303 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:06:57.277356 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:57.281254 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:57.282163 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:57.282172 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:57.282175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:57.282338 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:06:57.282836 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:06:57.282840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:06:57.282842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:06:57.283221 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:57.283608 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:57.283874 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 22:06:57.284122 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 22:06:57.285347 23555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:57.439584 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 23432
I20260927 22:06:57.465845 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 23071
I20260927 22:06:57.467559 23133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:06:57.598814 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 23071
2026-09-27T22:06:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5472 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T22:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:06:57Z Disabled control of system clock
I20260927 22:06:57.704116 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37389
--webserver_interface=127.3.159.62
--webserver_port=0
--builtin_ntp_servers=127.3.159.20:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:37389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:06:58.172657 23613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:58.173291 23613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:58.173527 23613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:58.190244 23613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:06:58.190477 23613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:58.190572 23613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:58.190702 23613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:06:58.190776 23613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:06:58.202894 23613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:37389
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:58.204686 23613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:58.207206 23613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:58.223374 23619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:58.223182 23618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:58.223541 23621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:58.225580 23613 server_base.cc:1034] running on GCE node
I20260927 22:06:58.226836 23613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:58.229003 23613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:58.230326 23613 hybrid_clock.cc:648] HybridClock initialized: now 1790546818230231 us; error 135 us; skew 500 ppm
I20260927 22:06:58.230832 23613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:06:58.234268 23613 webserver.cc:460] Webserver started at http://127.3.159.62:40349/ using document root <none> and password file <none>
I20260927 22:06:58.235289 23613 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:58.235458 23613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:58.235886 23613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:58.240229 23613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "8e8d4e2f32e44a058355544f5ffa4ba4"
format_stamp: "Formatted at 2026-09-27 22:06:58 on dist-test-slave-sg4j"
server_key: "fc9c2435ec3d5b0070c58412c62065d3"
server_key_iv: "e767395a74e6dcd929d6e43cc798514b"
server_key_version: "encryptionkey@0"
I20260927 22:06:58.241556 23613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "8e8d4e2f32e44a058355544f5ffa4ba4"
format_stamp: "Formatted at 2026-09-27 22:06:58 on dist-test-slave-sg4j"
server_key: "fc9c2435ec3d5b0070c58412c62065d3"
server_key_iv: "e767395a74e6dcd929d6e43cc798514b"
server_key_version: "encryptionkey@0"
I20260927 22:06:58.250979 23613 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.001s
I20260927 22:06:58.257130 23627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:58.259959 23613 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260927 22:06:58.260327 23613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "8e8d4e2f32e44a058355544f5ffa4ba4"
format_stamp: "Formatted at 2026-09-27 22:06:58 on dist-test-slave-sg4j"
server_key: "fc9c2435ec3d5b0070c58412c62065d3"
server_key_iv: "e767395a74e6dcd929d6e43cc798514b"
server_key_version: "encryptionkey@0"
I20260927 22:06:58.260687 23613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:58.280176 23613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:58.281569 23613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:58.281972 23613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:58.308998 23613 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:37389
I20260927 22:06:58.309039 23678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:37389 every 8 connection(s)
I20260927 22:06:58.311139 23613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:06:58.313446 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 23613
I20260927 22:06:58.313668 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:06:58.314270 3708 external_mini_cluster.cc:1428] Setting key d6b60e1fc617712a5aefae38ec0a4ff9
I20260927 22:06:58.317018 23679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:06:58.333359 23679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4: Bootstrap starting.
I20260927 22:06:58.333518 23682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:58.316802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.3.159.62:37389)
0927 22:06:58.318325 (+ 1523us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:58.318343 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:58.318406 (+ 63us) server_negotiation.cc:407] Connection header received
0927 22:06:58.320694 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:58.320730 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:58.321650 (+ 920us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:58.322252 (+ 602us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:58.323350 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:58.325190 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:58.326868 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:58.327915 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:58.328221 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:58.329894 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:58.329904 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:58.329930 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:58.330715 (+ 785us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:58.330830 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:58.330859 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:58.331060 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:58.331391 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 22:06:58.331913 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":893,"thread_start_us":593,"threads_started":1}
I20260927 22:06:58.338820 23679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4: Neither blocks nor log segments found. Creating new log.
I20260927 22:06:58.340075 23679 log.cc:824] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4: Log is configured to *not* fsync() on all Append() calls
I20260927 22:06:58.344430 23679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4: No bootstrap required, opened a new log
I20260927 22:06:58.354467 23679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8d4e2f32e44a058355544f5ffa4ba4" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37389 } }
I20260927 22:06:58.355090 23679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:06:58.355228 23679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e8d4e2f32e44a058355544f5ffa4ba4, State: Initialized, Role: FOLLOWER
I20260927 22:06:58.356104 23679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8d4e2f32e44a058355544f5ffa4ba4" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37389 } }
I20260927 22:06:58.356493 23679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:06:58.356709 23679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:06:58.356951 23679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:06:58.359959 23679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8d4e2f32e44a058355544f5ffa4ba4" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37389 } }
I20260927 22:06:58.360680 23679 leader_election.cc:302] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e8d4e2f32e44a058355544f5ffa4ba4; no voters:
I20260927 22:06:58.362022 23679 leader_election.cc:288] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:06:58.362280 23684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:06:58.363382 23684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [term 1 LEADER]: Becoming Leader. State: Replica: 8e8d4e2f32e44a058355544f5ffa4ba4, State: Running, Role: LEADER
I20260927 22:06:58.364264 23684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8d4e2f32e44a058355544f5ffa4ba4" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37389 } }
I20260927 22:06:58.365427 23679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:06:58.368327 23686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e8d4e2f32e44a058355544f5ffa4ba4. Latest consensus state: current_term: 1 leader_uuid: "8e8d4e2f32e44a058355544f5ffa4ba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8d4e2f32e44a058355544f5ffa4ba4" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37389 } } }
I20260927 22:06:58.368429 23685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e8d4e2f32e44a058355544f5ffa4ba4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e8d4e2f32e44a058355544f5ffa4ba4" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37389 } } }
I20260927 22:06:58.369580 23686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:58.369773 23685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4 [sys.catalog]: This master's current role is: LEADER
I20260927 22:06:58.375332 23691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:06:58.383196 23691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:06:58.397176 23691 catalog_manager.cc:1347] Generated new cluster ID: 265060f065fd451fa5be303d8800c697
I20260927 22:06:58.397418 23691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:06:58.418664 23691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:06:58.420322 23691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:06:58.433260 23691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4: Generated new TSK 0
I20260927 22:06:58.434577 23691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:06:58.442962 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.1:0
--local_ip_for_outbound_sockets=127.3.159.1
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37389
--builtin_ntp_servers=127.3.159.20:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 22:06:58.905860 23703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:58.906334 23703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:58.906507 23703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:58.916412 23703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:58.916568 23703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:58.916692 23703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.1
I20260927 22:06:58.930086 23703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.159.1
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.1
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:58.931551 23703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:58.933923 23703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:58.957823 23708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:58.959390 23711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:58.959499 23709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:58.960999 23703 server_base.cc:1034] running on GCE node
I20260927 22:06:58.962108 23703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:58.963835 23703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:58.965251 23703 hybrid_clock.cc:648] HybridClock initialized: now 1790546818965131 us; error 92 us; skew 500 ppm
I20260927 22:06:58.965831 23703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:06:58.970512 23703 webserver.cc:460] Webserver started at http://127.3.159.1:45683/ using document root <none> and password file <none>
I20260927 22:06:58.971716 23703 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:58.971916 23703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:58.972532 23703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:58.977085 23703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/data/instance:
uuid: "3a95efbea46f467b8af1c2e90ee0220b"
format_stamp: "Formatted at 2026-09-27 22:06:58 on dist-test-slave-sg4j"
server_key: "c1fbc386f70d31a3c7865e1921ea1e00"
server_key_iv: "1fdc8a59b3d5341ad29b873c04454f3b"
server_key_version: "encryptionkey@0"
I20260927 22:06:58.978478 23703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance:
uuid: "3a95efbea46f467b8af1c2e90ee0220b"
format_stamp: "Formatted at 2026-09-27 22:06:58 on dist-test-slave-sg4j"
server_key: "c1fbc386f70d31a3c7865e1921ea1e00"
server_key_iv: "1fdc8a59b3d5341ad29b873c04454f3b"
server_key_version: "encryptionkey@0"
I20260927 22:06:58.989319 23703 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s
I20260927 22:06:58.996438 23717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:58.999377 23703 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260927 22:06:58.999763 23703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/wal
uuid: "3a95efbea46f467b8af1c2e90ee0220b"
format_stamp: "Formatted at 2026-09-27 22:06:58 on dist-test-slave-sg4j"
server_key: "c1fbc386f70d31a3c7865e1921ea1e00"
server_key_iv: "1fdc8a59b3d5341ad29b873c04454f3b"
server_key_version: "encryptionkey@0"
I20260927 22:06:59.000160 23703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:59.041786 23703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:59.043182 23703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:59.043618 23703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:59.045290 23703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:59.047717 23703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:59.047902 23703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:59.048071 23703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:59.048168 23703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:59.100767 23703 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.1:39431
I20260927 22:06:59.100876 23829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.1:39431 every 8 connection(s)
I20260927 22:06:59.102782 23703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/data/info.pb
I20260927 22:06:59.103770 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 23703
I20260927 22:06:59.103993 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/wal/instance
I20260927 22:06:59.104667 3708 external_mini_cluster.cc:1428] Setting key ebd1e9acdd271b89edac74330bc0342a
I20260927 22:06:59.109862 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.2:0
--local_ip_for_outbound_sockets=127.3.159.2
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37389
--builtin_ntp_servers=127.3.159.20:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:06:59.122217 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:59.107235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:35771 (local address 127.3.159.62:37389)
0927 22:06:59.107920 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:59.107928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:59.110311 (+ 2383us) server_negotiation.cc:407] Connection header received
0927 22:06:59.112425 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:59.112432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:59.112552 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:59.112691 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:59.115207 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:59.116363 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:59.118676 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:59.119170 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:59.119374 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:59.121176 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:59.121181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:59.121185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:59.121220 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:59.121264 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:59.121278 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:59.121387 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:59.121711 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 22:06:59.121838 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":484,"threads_started":1}
I20260927 22:06:59.123425 23833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:59.108297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37389 (local address 127.3.159.1:35771)
0927 22:06:59.109897 (+ 1600us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:59.109958 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:59.112012 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:59.113263 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:59.113275 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:59.113745 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:59.114745 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:59.114780 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:59.116640 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:59.116645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:59.118278 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:59.118298 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:59.119510 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:59.119519 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:59.119814 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:59.120720 (+ 906us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:06:59.120722 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:59.120791 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:06:59.120895 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:59.121416 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:59.121431 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:59.121447 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 22:06:59.121679 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 22:06:59.122181 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":523,"threads_started":1}
I20260927 22:06:59.126170 23830 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37389
I20260927 22:06:59.126750 23830 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:59.128116 23830 heartbeater.cc:511] Master 127.3.159.62:37389 requested a full tablet report, sending...
I20260927 22:06:59.132261 23644 ts_manager.cc:194] Registered new tserver with Master: 3a95efbea46f467b8af1c2e90ee0220b (127.3.159.1:39431)
I20260927 22:06:59.135146 23644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.1:35771
W20260927 22:06:59.625836 23834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:06:59.626313 23834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:06:59.626426 23834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:06:59.636148 23834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:06:59.636394 23834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:06:59.636543 23834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.2
I20260927 22:06:59.649570 23834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.159.2
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.2
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:06:59.651415 23834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:06:59.654477 23834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:06:59.673781 23839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:59.673903 23840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:06:59.673959 23842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:06:59.674638 23834 server_base.cc:1034] running on GCE node
I20260927 22:06:59.675618 23834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:06:59.677098 23834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:06:59.678376 23834 hybrid_clock.cc:648] HybridClock initialized: now 1790546819678318 us; error 62 us; skew 500 ppm
I20260927 22:06:59.678864 23834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:06:59.683276 23834 webserver.cc:460] Webserver started at http://127.3.159.2:33519/ using document root <none> and password file <none>
I20260927 22:06:59.684468 23834 fs_manager.cc:359] Metadata directory not provided
I20260927 22:06:59.684613 23834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:06:59.685057 23834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:06:59.689543 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/data/instance:
uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e"
format_stamp: "Formatted at 2026-09-27 22:06:59 on dist-test-slave-sg4j"
server_key: "2b1d34fdc6bbd5f7a55dcdd627f75465"
server_key_iv: "cc91610d3d0146807b5d5aefca3932d2"
server_key_version: "encryptionkey@0"
I20260927 22:06:59.690864 23834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance:
uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e"
format_stamp: "Formatted at 2026-09-27 22:06:59 on dist-test-slave-sg4j"
server_key: "2b1d34fdc6bbd5f7a55dcdd627f75465"
server_key_iv: "cc91610d3d0146807b5d5aefca3932d2"
server_key_version: "encryptionkey@0"
I20260927 22:06:59.699172 23834 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260927 22:06:59.704979 23848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:59.707396 23834 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:06:59.707705 23834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/wal
uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e"
format_stamp: "Formatted at 2026-09-27 22:06:59 on dist-test-slave-sg4j"
server_key: "2b1d34fdc6bbd5f7a55dcdd627f75465"
server_key_iv: "cc91610d3d0146807b5d5aefca3932d2"
server_key_version: "encryptionkey@0"
I20260927 22:06:59.708086 23834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:06:59.754983 23834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:06:59.756239 23834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:06:59.756609 23834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:06:59.758204 23834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:06:59.760576 23834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:06:59.760762 23834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:59.760938 23834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:06:59.761114 23834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 22:06:59.816030 23834 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.2:38791
I20260927 22:06:59.816107 23960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.2:38791 every 8 connection(s)
I20260927 22:06:59.818061 23834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/data/info.pb
I20260927 22:06:59.828169 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 23834
I20260927 22:06:59.828435 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/wal/instance
I20260927 22:06:59.828979 3708 external_mini_cluster.cc:1428] Setting key 01371ed7ec91ffdd8f77e7fc0ddd7e4f
I20260927 22:06:59.833235 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.159.3:0
--local_ip_for_outbound_sockets=127.3.159.3
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37389
--builtin_ntp_servers=127.3.159.20:33947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 22:06:59.838469 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:59.822503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.2:32933 (local address 127.3.159.62:37389)
0927 22:06:59.823556 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0927 22:06:59.823562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:06:59.825485 (+ 1923us) server_negotiation.cc:407] Connection header received
0927 22:06:59.828258 (+ 2773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:06:59.828264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:06:59.828407 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:06:59.828559 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:06:59.830220 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:59.831269 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:59.833701 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:06:59.834225 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:06:59.834350 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:59.837248 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:06:59.837255 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:06:59.837259 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:06:59.837285 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:06:59.837345 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:06:59.837361 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:06:59.837577 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 22:06:59.838030 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 22:06:59.838155 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":527,"threads_started":1}
I20260927 22:06:59.840709 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:06:59.823624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37389 (local address 127.3.159.2:32933)
0927 22:06:59.825077 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0927 22:06:59.825138 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 22:06:59.827785 (+ 2647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:06:59.828876 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:06:59.828886 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:06:59.829012 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:06:59.829924 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:59.829955 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:59.831540 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:59.831546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:59.833335 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:06:59.833356 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:06:59.834658 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:06:59.834672 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:06:59.835233 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:06:59.836469 (+ 1236us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:06:59.836477 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:06:59.836613 (+ 136us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:06:59.836815 (+ 202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:06:59.837623 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:06:59.837640 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:06:59.837655 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 22:06:59.837947 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 22:06:59.838670 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":436,"threads_started":1}
I20260927 22:06:59.843333 23961 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37389
I20260927 22:06:59.844126 23961 heartbeater.cc:464] Registering TS with master...
I20260927 22:06:59.845237 23961 heartbeater.cc:511] Master 127.3.159.62:37389 requested a full tablet report, sending...
I20260927 22:06:59.848177 23644 ts_manager.cc:194] Registered new tserver with Master: 0d6547c4c80e4eb9b0caad7e6fbcf46e (127.3.159.2:38791)
I20260927 22:06:59.849326 23644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.2:32933
I20260927 22:07:00.142253 23830 heartbeater.cc:503] Master 127.3.159.62:37389 was elected leader, sending a full tablet report...
W20260927 22:07:00.332589 23965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:00.333341 23965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:00.333570 23965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:00.346343 23965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:00.346695 23965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:00.346909 23965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.159.3
I20260927 22:07:00.363019 23965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:33947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.159.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.159.3
--webserver_port=0
--tserver_master_addrs=127.3.159.62:37389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.159.3
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:00.365412 23965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:00.367995 23965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:00.391177 23971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:00.391043 23970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:00.391116 23973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:00.393277 23965 server_base.cc:1034] running on GCE node
I20260927 22:07:00.394409 23965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:00.396533 23965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:00.398128 23965 hybrid_clock.cc:648] HybridClock initialized: now 1790546820398057 us; error 75 us; skew 500 ppm
I20260927 22:07:00.398634 23965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 22:07:00.402573 23965 webserver.cc:460] Webserver started at http://127.3.159.3:45927/ using document root <none> and password file <none>
I20260927 22:07:00.403661 23965 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:00.403800 23965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:00.404222 23965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:00.408974 23965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/data/instance:
uuid: "3c13d22f9a0b4f6287bde5a5d80200f9"
format_stamp: "Formatted at 2026-09-27 22:07:00 on dist-test-slave-sg4j"
server_key: "db5430f107370068ee6fd0bb9088a8d1"
server_key_iv: "456de68430f128bb86672b83be58f363"
server_key_version: "encryptionkey@0"
I20260927 22:07:00.410714 23965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance:
uuid: "3c13d22f9a0b4f6287bde5a5d80200f9"
format_stamp: "Formatted at 2026-09-27 22:07:00 on dist-test-slave-sg4j"
server_key: "db5430f107370068ee6fd0bb9088a8d1"
server_key_iv: "456de68430f128bb86672b83be58f363"
server_key_version: "encryptionkey@0"
I20260927 22:07:00.420173 23965 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.007s
I20260927 22:07:00.426115 23979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:00.428743 23965 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260927 22:07:00.429173 23965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/wal
uuid: "3c13d22f9a0b4f6287bde5a5d80200f9"
format_stamp: "Formatted at 2026-09-27 22:07:00 on dist-test-slave-sg4j"
server_key: "db5430f107370068ee6fd0bb9088a8d1"
server_key_iv: "456de68430f128bb86672b83be58f363"
server_key_version: "encryptionkey@0"
I20260927 22:07:00.429719 23965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:00.455320 23965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:00.456555 23965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:00.456939 23965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:00.458709 23965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 22:07:00.461088 23965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 22:07:00.461261 23965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:00.461458 23965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 22:07:00.461577 23965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 22:07:00.509526 23965 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.3:42729
I20260927 22:07:00.509618 24091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.3:42729 every 8 connection(s)
I20260927 22:07:00.511408 23965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/data/info.pb
I20260927 22:07:00.513767 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 23965
I20260927 22:07:00.513993 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/wal/instance
I20260927 22:07:00.514513 3708 external_mini_cluster.cc:1428] Setting key f17e1adb2d1d2a42c445fa91baa282fb
I20260927 22:07:00.531948 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.515775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.3:45767 (local address 127.3.159.62:37389)
0927 22:07:00.516658 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:00.516663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:00.518880 (+ 2217us) server_negotiation.cc:407] Connection header received
0927 22:07:00.521070 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:00.521076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:00.521167 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:00.521305 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:00.523362 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.524167 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.526580 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.527152 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.527324 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.529802 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:00.529809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:00.529813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:00.529837 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:00.529892 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:00.529909 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:00.530003 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:00.531017 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0927 22:07:00.531181 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":580,"threads_started":1}
I20260927 22:07:00.533455 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.516885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37389 (local address 127.3.159.3:45767)
0927 22:07:00.518514 (+ 1629us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:00.518584 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:00.520589 (+ 2005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:00.521871 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:00.521883 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:00.522031 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:00.523006 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.523039 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.524407 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.524412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.526280 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.526305 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.527605 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.527615 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.527965 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.529090 (+ 1125us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 22:07:00.529093 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:00.529160 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 22:07:00.529330 (+ 170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:00.530067 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:00.530083 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:00.530094 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 22:07:00.530420 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 22:07:00.531210 (+ 790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":545,"threads_started":1}
I20260927 22:07:00.535920 24092 heartbeater.cc:347] Connected to a master server at 127.3.159.62:37389
I20260927 22:07:00.536576 24092 heartbeater.cc:464] Registering TS with master...
I20260927 22:07:00.538130 24092 heartbeater.cc:511] Master 127.3.159.62:37389 requested a full tablet report, sending...
I20260927 22:07:00.540541 23644 ts_manager.cc:194] Registered new tserver with Master: 3c13d22f9a0b4f6287bde5a5d80200f9 (127.3.159.3:42729)
I20260927 22:07:00.541846 23644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.159.3:45767
I20260927 22:07:00.548079 3708 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 22:07:00.563856 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.555242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.3.159.62:37389)
0927 22:07:00.555599 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:00.555605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:00.556210 (+ 605us) server_negotiation.cc:407] Connection header received
0927 22:07:00.556336 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:00.556341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:00.556449 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:00.556579 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:00.557746 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.558622 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.560502 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.561092 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.561261 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.562930 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:00.562937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:00.562941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:00.562968 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:00.563014 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:00.563029 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:00.563149 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:00.563429 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 22:07:00.563565 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260927 22:07:00.570490 23644 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 22:07:00.580147 23644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 22:07:00.615010 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.600434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:42729 (local address 127.0.0.1:52018)
0927 22:07:00.601105 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:00.601130 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:00.601288 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:00.603275 (+ 1987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:00.603289 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:00.603346 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:07:00.604022 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.604044 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.607681 (+ 3637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.607694 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.611132 (+ 3438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.611166 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.613638 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.613650 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.613962 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.614056 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 22:07:00.614249 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 22:07:00.614537 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":5,"thread_start_us":363,"threads_started":1}
I20260927 22:07:00.615105 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.600811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.3.159.3:42729)
0927 22:07:00.602106 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:00.602122 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:00.602179 (+ 57us) server_negotiation.cc:407] Connection header received
0927 22:07:00.602341 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:00.602353 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:00.602828 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:00.604360 (+ 1532us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:07:00.604821 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.607338 (+ 2517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.611470 (+ 4132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.613256 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.613505 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.613879 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:00.614378 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 22:07:00.614635 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1008,"thread_start_us":780,"threads_started":1}
I20260927 22:07:00.618445 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.597810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:38791 (local address 127.0.0.1:59630)
0927 22:07:00.599147 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:00.599213 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:00.599467 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:00.604146 (+ 4679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:00.604157 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:00.604213 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:07:00.604866 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.604888 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.610246 (+ 5358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.610257 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.613791 (+ 3534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.613810 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.617305 (+ 3495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.617313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.617637 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.617688 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 22:07:00.617812 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 22:07:00.617983 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":59,"thread_start_us":427,"threads_started":1}
I20260927 22:07:00.619558 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.597929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.3.159.2:38791)
0927 22:07:00.600308 (+ 2379us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:00.600321 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:00.600367 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:07:00.600680 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:00.600696 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:00.601210 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:00.603697 (+ 2487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:07:00.605224 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.609940 (+ 4716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.614068 (+ 4128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.616701 (+ 2633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.616845 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.617141 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:00.618828 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0927 22:07:00.619102 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2160,"thread_start_us":1969,"threads_started":1}
I20260927 22:07:00.621344 24027 tablet_service.cc:1467] Processing CreateTablet for tablet dceefc32780f49dba9ff01a1cd36ea71 (DEFAULT_TABLE table=test-table [id=183692fe193740fc851d0bd8ebb630c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:07:00.622449 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.602132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.1:39431 (local address 127.0.0.1:34576)
0927 22:07:00.611271 (+ 9139us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:00.611296 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:00.611462 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:00.613019 (+ 1557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:00.613028 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:00.613069 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:07:00.613652 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.613669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.617207 (+ 3538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.617214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.619358 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.619376 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.621288 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.621301 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.621605 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.621652 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 22:07:00.621798 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 22:07:00.621981 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8914,"thread_start_us":8662,"threads_started":1}
I20260927 22:07:00.623191 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.601656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34576 (local address 127.3.159.1:39431)
0927 22:07:00.610702 (+ 9046us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:00.610713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:00.611607 (+ 894us) server_negotiation.cc:407] Connection header received
0927 22:07:00.611895 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:00.611905 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:00.612402 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:00.612725 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:07:00.614008 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.616602 (+ 2594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.619593 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.620984 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.621154 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.621550 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:00.621944 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 22:07:00.622153 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8786,"thread_start_us":8568,"threads_started":1}
I20260927 22:07:00.623493 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dceefc32780f49dba9ff01a1cd36ea71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:00.624820 23896 tablet_service.cc:1467] Processing CreateTablet for tablet dceefc32780f49dba9ff01a1cd36ea71 (DEFAULT_TABLE table=test-table [id=183692fe193740fc851d0bd8ebb630c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:07:00.626484 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dceefc32780f49dba9ff01a1cd36ea71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:00.628765 23765 tablet_service.cc:1467] Processing CreateTablet for tablet dceefc32780f49dba9ff01a1cd36ea71 (DEFAULT_TABLE table=test-table [id=183692fe193740fc851d0bd8ebb630c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 22:07:00.631162 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dceefc32780f49dba9ff01a1cd36ea71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:00.647361 24112 tablet_bootstrap.cc:492] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: Bootstrap starting.
I20260927 22:07:00.650477 24111 tablet_bootstrap.cc:492] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: Bootstrap starting.
I20260927 22:07:00.652922 24112 tablet_bootstrap.cc:654] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:00.654917 24112 log.cc:824] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:00.654995 24113 tablet_bootstrap.cc:492] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: Bootstrap starting.
I20260927 22:07:00.655391 24111 tablet_bootstrap.cc:654] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:00.657016 24111 log.cc:824] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:00.661170 24112 tablet_bootstrap.cc:492] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: No bootstrap required, opened a new log
I20260927 22:07:00.661456 24113 tablet_bootstrap.cc:654] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:00.661654 24112 ts_tablet_manager.cc:1397] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260927 22:07:00.662276 24111 tablet_bootstrap.cc:492] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: No bootstrap required, opened a new log
I20260927 22:07:00.662895 24111 ts_tablet_manager.cc:1397] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260927 22:07:00.663794 24113 log.cc:824] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:00.669688 24113 tablet_bootstrap.cc:492] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: No bootstrap required, opened a new log
I20260927 22:07:00.670172 24113 ts_tablet_manager.cc:1397] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260927 22:07:00.673906 24112 raft_consensus.cc:348] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38791 } } peers { permanent_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42729 } } peers { permanent_uuid: "3a95efbea46f467b8af1c2e90ee0220b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 39431 } }
I20260927 22:07:00.674407 24112 raft_consensus.cc:374] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:00.674499 24112 raft_consensus.cc:729] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6547c4c80e4eb9b0caad7e6fbcf46e, State: Initialized, Role: FOLLOWER
I20260927 22:07:00.675217 24112 consensus_queue.cc:260] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38791 } } peers { permanent_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42729 } } peers { permanent_uuid: "3a95efbea46f467b8af1c2e90ee0220b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 39431 } }
I20260927 22:07:00.676489 23961 heartbeater.cc:503] Master 127.3.159.62:37389 was elected leader, sending a full tablet report...
I20260927 22:07:00.676322 24111 raft_consensus.cc:348] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38791 } } peers { permanent_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42729 } } peers { permanent_uuid: "3a95efbea46f467b8af1c2e90ee0220b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 39431 } }
I20260927 22:07:00.676970 24111 raft_consensus.cc:374] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:00.677080 24111 raft_consensus.cc:729] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c13d22f9a0b4f6287bde5a5d80200f9, State: Initialized, Role: FOLLOWER
I20260927 22:07:00.677304 24112 ts_tablet_manager.cc:1428] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260927 22:07:00.678002 24111 consensus_queue.cc:260] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38791 } } peers { permanent_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42729 } } peers { permanent_uuid: "3a95efbea46f467b8af1c2e90ee0220b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 39431 } }
I20260927 22:07:00.679608 24092 heartbeater.cc:503] Master 127.3.159.62:37389 was elected leader, sending a full tablet report...
I20260927 22:07:00.680634 24111 ts_tablet_manager.cc:1428] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: Time spent starting tablet: real 0.018s user 0.012s sys 0.006s
I20260927 22:07:00.682685 24113 raft_consensus.cc:348] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38791 } } peers { permanent_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42729 } } peers { permanent_uuid: "3a95efbea46f467b8af1c2e90ee0220b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 39431 } }
I20260927 22:07:00.683629 24113 raft_consensus.cc:374] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:00.683806 24113 raft_consensus.cc:729] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a95efbea46f467b8af1c2e90ee0220b, State: Initialized, Role: FOLLOWER
I20260927 22:07:00.684793 24113 consensus_queue.cc:260] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38791 } } peers { permanent_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42729 } } peers { permanent_uuid: "3a95efbea46f467b8af1c2e90ee0220b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 39431 } }
I20260927 22:07:00.687786 24113 ts_tablet_manager.cc:1428] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
W20260927 22:07:00.764016 24093 tablet.cc:2369] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:07:00.764928 24119 raft_consensus.cc:482] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 22:07:00.765492 24119 raft_consensus.cc:504] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38791 } } peers { permanent_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42729 } } peers { permanent_uuid: "3a95efbea46f467b8af1c2e90ee0220b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 39431 } }
I20260927 22:07:00.769791 24119 leader_election.cc:288] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d6547c4c80e4eb9b0caad7e6fbcf46e (127.3.159.2:38791), 3c13d22f9a0b4f6287bde5a5d80200f9 (127.3.159.3:42729)
I20260927 22:07:00.780280 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.770223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:54939 (local address 127.3.159.2:38791)
0927 22:07:00.770614 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:00.770620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:00.771346 (+ 726us) server_negotiation.cc:407] Connection header received
0927 22:07:00.771566 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:00.771572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:00.771693 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:00.771842 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:07:00.773105 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.774390 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.777547 (+ 3157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.778691 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.778825 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.778925 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:00.779629 (+ 704us) server_negotiation.cc:299] Negotiation successful
0927 22:07:00.779781 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260927 22:07:00.780229 24120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.770170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.2:38791 (local address 127.3.159.1:54939)
0927 22:07:00.771122 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:00.771163 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:00.771404 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:00.772146 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:00.772154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:00.772208 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:07:00.772755 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.772770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.774668 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.774676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.777126 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.777153 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.779054 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.779066 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.779309 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.779358 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 22:07:00.779525 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 22:07:00.779709 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":4,"thread_start_us":522,"threads_started":1}
I20260927 22:07:00.782518 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.771827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.159.1:34157 (local address 127.3.159.3:42729)
0927 22:07:00.772124 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:00.772130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:00.772423 (+ 293us) server_negotiation.cc:407] Connection header received
0927 22:07:00.772704 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:00.772712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:00.772799 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:00.772921 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 22:07:00.774182 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.775429 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.778344 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.780210 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.780902 (+ 692us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.781076 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:00.781868 (+ 792us) server_negotiation.cc:299] Negotiation successful
0927 22:07:00.782157 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260927 22:07:00.782889 24121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.771486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.3:42729 (local address 127.3.159.1:34157)
0927 22:07:00.772250 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:00.772309 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:00.772465 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:00.773357 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:00.773363 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:00.773432 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 22:07:00.773877 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.773891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.775802 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.775807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.777992 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:00.778014 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.781313 (+ 3299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.781321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:00.781592 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.781657 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 22:07:00.781779 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 22:07:00.781934 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":36,"thread_start_us":370,"threads_started":1}
I20260927 22:07:00.783579 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dceefc32780f49dba9ff01a1cd36ea71" candidate_uuid: "3a95efbea46f467b8af1c2e90ee0220b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" is_pre_election: true
I20260927 22:07:00.784678 23916 raft_consensus.cc:2454] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a95efbea46f467b8af1c2e90ee0220b in term 0.
I20260927 22:07:00.784698 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dceefc32780f49dba9ff01a1cd36ea71" candidate_uuid: "3a95efbea46f467b8af1c2e90ee0220b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" is_pre_election: true
I20260927 22:07:00.785502 24047 raft_consensus.cc:2454] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a95efbea46f467b8af1c2e90ee0220b in term 0.
I20260927 22:07:00.786191 23719 leader_election.cc:302] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6547c4c80e4eb9b0caad7e6fbcf46e, 3a95efbea46f467b8af1c2e90ee0220b; no voters:
I20260927 22:07:00.786913 24119 raft_consensus.cc:2781] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 22:07:00.787168 24119 raft_consensus.cc:482] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 22:07:00.787343 24119 raft_consensus.cc:3037] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:00.791385 24119 raft_consensus.cc:504] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38791 } } peers { permanent_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42729 } } peers { permanent_uuid: "3a95efbea46f467b8af1c2e90ee0220b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 39431 } }
I20260927 22:07:00.792243 24119 leader_election.cc:288] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [CANDIDATE]: Term 1 election: Requested vote from peers 0d6547c4c80e4eb9b0caad7e6fbcf46e (127.3.159.2:38791), 3c13d22f9a0b4f6287bde5a5d80200f9 (127.3.159.3:42729)
I20260927 22:07:00.793249 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dceefc32780f49dba9ff01a1cd36ea71" candidate_uuid: "3a95efbea46f467b8af1c2e90ee0220b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e"
I20260927 22:07:00.793346 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dceefc32780f49dba9ff01a1cd36ea71" candidate_uuid: "3a95efbea46f467b8af1c2e90ee0220b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9"
I20260927 22:07:00.793658 23916 raft_consensus.cc:3037] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:00.793828 24047 raft_consensus.cc:3037] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:00.796715 24047 raft_consensus.cc:2454] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a95efbea46f467b8af1c2e90ee0220b in term 1.
I20260927 22:07:00.796720 23916 raft_consensus.cc:2454] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a95efbea46f467b8af1c2e90ee0220b in term 1.
I20260927 22:07:00.797639 23719 leader_election.cc:302] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6547c4c80e4eb9b0caad7e6fbcf46e, 3a95efbea46f467b8af1c2e90ee0220b; no voters:
I20260927 22:07:00.798116 24119 raft_consensus.cc:2781] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:00.798754 24119 raft_consensus.cc:686] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 1 LEADER]: Becoming Leader. State: Replica: 3a95efbea46f467b8af1c2e90ee0220b, State: Running, Role: LEADER
I20260927 22:07:00.799613 24119 consensus_queue.cc:237] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38791 } } peers { permanent_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42729 } } peers { permanent_uuid: "3a95efbea46f467b8af1c2e90ee0220b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 39431 } }
I20260927 22:07:00.808269 23644 catalog_manager.cc:5495] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a95efbea46f467b8af1c2e90ee0220b (127.3.159.1). New cstate: current_term: 1 leader_uuid: "3a95efbea46f467b8af1c2e90ee0220b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a95efbea46f467b8af1c2e90ee0220b" member_type: VOTER last_known_addr { host: "127.3.159.1" port: 39431 } health_report { overall_health: HEALTHY } } }
W20260927 22:07:00.821902 23962 tablet.cc:2369] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 22:07:00.857340 23831 tablet.cc:2369] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:07:00.870949 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.862531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.3.159.1:39431)
0927 22:07:00.862885 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:00.862890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:00.862922 (+ 32us) server_negotiation.cc:407] Connection header received
0927 22:07:00.863054 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:00.863059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:00.863172 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:00.863367 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:07:00.864029 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.865160 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.867489 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.868051 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.868195 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.869074 (+ 879us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:07:00.869842 (+ 768us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:07:00.870039 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:00.870341 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 22:07:00.870472 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260927 22:07:00.883749 24047 raft_consensus.cc:1261] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9 [term 1 FOLLOWER]: Refusing update from remote peer 3a95efbea46f467b8af1c2e90ee0220b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:07:00.884047 23916 raft_consensus.cc:1261] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e [term 1 FOLLOWER]: Refusing update from remote peer 3a95efbea46f467b8af1c2e90ee0220b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:07:00.885802 24122 consensus_queue.cc:1035] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6547c4c80e4eb9b0caad7e6fbcf46e" member_type: VOTER last_known_addr { host: "127.3.159.2" port: 38791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260927 22:07:00.886874 24119 consensus_queue.cc:1035] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c13d22f9a0b4f6287bde5a5d80200f9" member_type: VOTER last_known_addr { host: "127.3.159.3" port: 42729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:07:00.938315 23643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:07:00.938756 23643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 22:07:00.944578 23643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e8d4e2f32e44a058355544f5ffa4ba4: Sending DeleteTablet for 3 replicas of tablet dceefc32780f49dba9ff01a1cd36ea71
I20260927 22:07:00.946408 23896 tablet_service.cc:1514] Processing DeleteTablet for tablet dceefc32780f49dba9ff01a1cd36ea71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:07:00 UTC) from {username='slave'} at 127.0.0.1:59630
I20260927 22:07:00.946446 24026 tablet_service.cc:1514] Processing DeleteTablet for tablet dceefc32780f49dba9ff01a1cd36ea71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:07:00 UTC) from {username='slave'} at 127.0.0.1:52018
I20260927 22:07:00.946693 23765 tablet_service.cc:1514] Processing DeleteTablet for tablet dceefc32780f49dba9ff01a1cd36ea71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:07:00 UTC) from {username='slave'} at 127.0.0.1:34576
I20260927 22:07:00.947332 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet dceefc32780f49dba9ff01a1cd36ea71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 22:07:00 UTC) from {username='slave'} at 127.0.0.1:52018
I20260927 22:07:00.948786 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 23703
I20260927 22:07:00.951323 24135 tablet_replica.cc:330] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: stopping tablet replica
I20260927 22:07:00.952406 24135 raft_consensus.cc:2229] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:07:00.953604 24135 raft_consensus.cc:2258] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:07:00.954955 24137 tablet_replica.cc:330] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: stopping tablet replica
I20260927 22:07:00.956068 24136 tablet_replica.cc:330] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: stopping tablet replica
I20260927 22:07:00.956934 24137 raft_consensus.cc:2229] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 1 LEADER]: Raft consensus shutting down.
I20260927 22:07:00.957803 24135 ts_tablet_manager.cc:1905] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:07:00.958190 24136 raft_consensus.cc:2229] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 22:07:00.959169 24136 raft_consensus.cc:2258] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:07:00.960736 24137 raft_consensus.cc:2258] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 22:07:00.965087 24136 ts_tablet_manager.cc:1905] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:07:00.968026 24137 ts_tablet_manager.cc:1905] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 22:07:00.975572 24136 ts_tablet_manager.cc:1918] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:07:00.975842 24136 log.cc:1196] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-1/wal/wals/dceefc32780f49dba9ff01a1cd36ea71
I20260927 22:07:00.976498 24136 ts_tablet_manager.cc:1939] T dceefc32780f49dba9ff01a1cd36ea71 P 0d6547c4c80e4eb9b0caad7e6fbcf46e: Deleting consensus metadata
I20260927 22:07:00.978197 24135 ts_tablet_manager.cc:1918] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:07:00.978564 24135 log.cc:1196] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-2/wal/wals/dceefc32780f49dba9ff01a1cd36ea71
I20260927 22:07:00.979240 24135 ts_tablet_manager.cc:1939] T dceefc32780f49dba9ff01a1cd36ea71 P 3c13d22f9a0b4f6287bde5a5d80200f9: Deleting consensus metadata
I20260927 22:07:00.979210 23629 catalog_manager.cc:4841] TS 0d6547c4c80e4eb9b0caad7e6fbcf46e (127.3.159.2:38791): tablet dceefc32780f49dba9ff01a1cd36ea71 (table test-table [id=183692fe193740fc851d0bd8ebb630c7]) successfully deleted
I20260927 22:07:00.980847 24137 ts_tablet_manager.cc:1918] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 22:07:00.981098 24137 log.cc:1196] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: Deleting WAL directory at /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790546447244210-3708-0/minicluster-data/ts-0/wal/wals/dceefc32780f49dba9ff01a1cd36ea71
I20260927 22:07:00.981809 24137 ts_tablet_manager.cc:1939] T dceefc32780f49dba9ff01a1cd36ea71 P 3a95efbea46f467b8af1c2e90ee0220b: Deleting consensus metadata
I20260927 22:07:00.981904 23629 catalog_manager.cc:4841] TS 3c13d22f9a0b4f6287bde5a5d80200f9 (127.3.159.3:42729): tablet dceefc32780f49dba9ff01a1cd36ea71 (table test-table [id=183692fe193740fc851d0bd8ebb630c7]) successfully deleted
I20260927 22:07:00.983309 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:00.950392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.3.159.1:39431)
0927 22:07:00.950716 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:00.950854 (+ 138us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:00.968135 (+ 17281us) server_negotiation.cc:407] Connection header received
0927 22:07:00.968786 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:00.968794 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:00.968940 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:00.969031 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:00.969914 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.971261 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.973626 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:00.974350 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:00.974480 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:00.980784 (+ 6304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:00.980802 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:00.980839 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:00.981985 (+ 1146us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:00.982085 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:00.982130 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:00.982248 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:00.982627 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 22:07:00.982780 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 22:07:00.983237 23629 catalog_manager.cc:4808] TS 3c13d22f9a0b4f6287bde5a5d80200f9 (127.3.159.3:42729): delete failed for tablet dceefc32780f49dba9ff01a1cd36ea71 because the tablet was not found. No further retry: Not found: Tablet not found: dceefc32780f49dba9ff01a1cd36ea71
I20260927 22:07:00.984308 23629 catalog_manager.cc:4841] TS 3a95efbea46f467b8af1c2e90ee0220b (127.3.159.1:39431): tablet dceefc32780f49dba9ff01a1cd36ea71 (table test-table [id=183692fe193740fc851d0bd8ebb630c7]) successfully deleted
I20260927 22:07:00.984866 23825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:01.102947 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 23703
I20260927 22:07:01.128111 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 23834
I20260927 22:07:01.137665 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:01.129492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.3.159.2:38791)
0927 22:07:01.129832 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:01.129837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:01.129863 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:07:01.129990 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:01.129994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:01.130111 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:01.130218 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:01.130842 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:01.131644 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:01.133315 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:01.133936 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:01.134078 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:01.135673 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:01.135716 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:01.135734 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:01.136446 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:01.136510 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:01.136535 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:01.136705 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:01.136979 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 22:07:01.137166 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260927 22:07:01.138715 23956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:01.254474 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 23834
I20260927 22:07:01.277990 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 23965
I20260927 22:07:01.287786 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:01.279267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.3.159.3:42729)
0927 22:07:01.279549 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:01.279554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:01.279577 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:07:01.279793 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:01.279797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:01.279897 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:01.280010 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:01.280901 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:01.281804 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:01.283442 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:01.283945 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:01.284104 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:01.285920 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:01.285932 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:01.285949 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:01.286735 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:01.286806 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:01.286831 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:01.286969 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:01.287318 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 22:07:01.287450 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 22:07:01.289180 24087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:01.422194 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 23965
I20260927 22:07:01.443054 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 23613
I20260927 22:07:01.444824 23674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:01.578807 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 23613
2026-09-27T22:07:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3974 ms)
[----------] 6 tests from EncryptionPolicyTest (28730 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:01 dist-test-slave-sg4j krb5kdc[24145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:01 dist-test-slave-sg4j krb5kdc[24145](info): set up 2 sockets
Sep 27 22:07:01 dist-test-slave-sg4j krb5kdc[24145](info): commencing operation
krb5kdc: starting...
W20260927 22:07:02.786370 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.003s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:03 dist-test-slave-sg4j krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:03.123225 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43433
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:03.574123 24158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:03.574568 24158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:03.574760 24158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:03.583581 24158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:03.583768 24158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:03.583822 24158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:03.583974 24158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:03.584055 24158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:03.596489 24158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43433
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:03.598415 24158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:03.600440 24158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:03.615994 24163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:03.616298 24164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:03.616200 24166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:03.618063 24158 server_base.cc:1034] running on GCE node
I20260927 22:07:03.619153 24158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:03.621016 24158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:03.622499 24158 hybrid_clock.cc:648] HybridClock initialized: now 1790546823622390 us; error 74 us; skew 500 ppm
I20260927 22:07:03.623010 24158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:03 dist-test-slave-sg4j krb5kdc[24145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:03.626999 24158 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:03.629041 24158 webserver.cc:460] Webserver started at http://127.0.0.1:38313/ using document root <none> and password file <none>
I20260927 22:07:03.629984 24158 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:03.630143 24158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:03.630492 24158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:03.634090 24158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "b98f3f2b355747cfafea7ed2e0b4b177"
format_stamp: "Formatted at 2026-09-27 22:07:03 on dist-test-slave-sg4j"
server_key: "c9a0fa15a1d10c04caa2e79c6626cda7"
server_key_iv: "02aecda2c3a00861981e68d45f9dd12a"
server_key_version: "encryptionkey@0"
I20260927 22:07:03.634964 24158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "b98f3f2b355747cfafea7ed2e0b4b177"
format_stamp: "Formatted at 2026-09-27 22:07:03 on dist-test-slave-sg4j"
server_key: "c9a0fa15a1d10c04caa2e79c6626cda7"
server_key_iv: "02aecda2c3a00861981e68d45f9dd12a"
server_key_version: "encryptionkey@0"
I20260927 22:07:03.642089 24158 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:07:03.647617 24173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:03.649958 24158 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260927 22:07:03.650254 24158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "b98f3f2b355747cfafea7ed2e0b4b177"
format_stamp: "Formatted at 2026-09-27 22:07:03 on dist-test-slave-sg4j"
server_key: "c9a0fa15a1d10c04caa2e79c6626cda7"
server_key_iv: "02aecda2c3a00861981e68d45f9dd12a"
server_key_version: "encryptionkey@0"
I20260927 22:07:03.650525 24158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:03.683328 24158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:03.690178 24158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:03.690629 24158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:03.719336 24158 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43433
I20260927 22:07:03.719394 24224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43433 every 8 connection(s)
I20260927 22:07:03.721294 24158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:03.726605 24225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:03.729525 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 24158
I20260927 22:07:03.729789 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:03.730672 3708 external_mini_cluster.cc:1428] Setting key e38ad03f8bfb262ee088cdb64c0ce78d
I20260927 22:07:03.742357 24225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177: Bootstrap starting.
I20260927 22:07:03.746719 24225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:03.748186 24225 log.cc:824] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:07:03 dist-test-slave-sg4j krb5kdc[24145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:07:03.752197 24225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177: No bootstrap required, opened a new log
I20260927 22:07:03.762621 24225 raft_consensus.cc:348] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98f3f2b355747cfafea7ed2e0b4b177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } }
I20260927 22:07:03.763145 24225 raft_consensus.cc:374] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:03.763309 24225 raft_consensus.cc:729] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b98f3f2b355747cfafea7ed2e0b4b177, State: Initialized, Role: FOLLOWER
I20260927 22:07:03.763991 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:03.733686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53639 (local address 127.0.0.1:43433)
0927 22:07:03.734868 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:03.734886 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:03.734952 (+ 66us) server_negotiation.cc:407] Connection header received
0927 22:07:03.737238 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:03.737279 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:03.738115 (+ 836us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:03.738669 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:03.740740 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:03.742713 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:03.745031 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:03.746072 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:03.746265 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:03.752007 (+ 5742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:03.752018 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:03.752043 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:03.752100 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:03.756783 (+ 4683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:03.758269 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:03.758279 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:03.758289 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:03.758658 (+ 369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:03.759368 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:03.759373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:03.759375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:03.760320 (+ 945us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:03.760850 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:03.761621 (+ 771us) server_negotiation.cc:299] Negotiation successful
0927 22:07:03.762203 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":566,"threads_started":1}
I20260927 22:07:03.764034 24225 consensus_queue.cc:260] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98f3f2b355747cfafea7ed2e0b4b177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } }
I20260927 22:07:03.764401 24225 raft_consensus.cc:388] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:03.764525 24225 raft_consensus.cc:482] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:03.764675 24225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:03.768054 24225 raft_consensus.cc:504] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98f3f2b355747cfafea7ed2e0b4b177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } }
I20260927 22:07:03.768779 24225 leader_election.cc:302] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b98f3f2b355747cfafea7ed2e0b4b177; no voters:
I20260927 22:07:03.769626 24225 leader_election.cc:288] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:03.769871 24230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:03.770608 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:03.771472 24230 raft_consensus.cc:686] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [term 1 LEADER]: Becoming Leader. State: Replica: b98f3f2b355747cfafea7ed2e0b4b177, State: Running, Role: LEADER
I20260927 22:07:03.772531 24225 sys_catalog.cc:564] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:03.772410 24230 consensus_queue.cc:237] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98f3f2b355747cfafea7ed2e0b4b177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } }
I20260927 22:07:03.777906 24231 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b98f3f2b355747cfafea7ed2e0b4b177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98f3f2b355747cfafea7ed2e0b4b177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } } }
I20260927 22:07:03.778769 24231 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:03.779696 24232 sys_catalog.cc:455] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b98f3f2b355747cfafea7ed2e0b4b177. Latest consensus state: current_term: 1 leader_uuid: "b98f3f2b355747cfafea7ed2e0b4b177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b98f3f2b355747cfafea7ed2e0b4b177" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43433 } } }
I20260927 22:07:03.780282 24232 sys_catalog.cc:458] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:03.785997 24242 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:03.794116 24242 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:03.809636 24242 catalog_manager.cc:1347] Generated new cluster ID: 3c7fcc16c8e546af9c9a7e84bf63cc62
I20260927 22:07:03.809796 24242 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:03.811046 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:03.787450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47695 (local address 127.0.0.1:43433)
0927 22:07:03.787920 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:03.787928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:03.787963 (+ 35us) server_negotiation.cc:407] Connection header received
0927 22:07:03.788232 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:03.788245 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:03.788359 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:03.788536 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:03.790753 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:03.791791 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:03.793847 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:03.794393 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:03.794534 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:03.799465 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:03.799477 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:03.799486 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:03.799570 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:03.805176 (+ 5606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:03.806822 (+ 1646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:03.806833 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:03.806836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:03.807053 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:03.807770 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:03.807777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:03.807784 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:03.808452 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:03.809273 (+ 821us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:03.810342 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0927 22:07:03.810609 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298}
I20260927 22:07:03.828095 24242 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:03.830734 24242 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:03.857059 24242 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b98f3f2b355747cfafea7ed2e0b4b177: Generated new TSK 0
I20260927 22:07:03.858366 24242 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:03.870193 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24158
I20260927 22:07:03.872215 24220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:03.980137 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 24158
W20260927 22:07:04.001291 24143 connection.cc:535] client connection to 127.0.0.1:43433 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43433): Connection reset by peer (error 104)
2026-09-27T22:07:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:04 dist-test-slave-sg4j krb5kdc[24257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:04 dist-test-slave-sg4j krb5kdc[24257](info): set up 2 sockets
Sep 27 22:07:04 dist-test-slave-sg4j krb5kdc[24257](info): commencing operation
krb5kdc: starting...
W20260927 22:07:05.158809 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:05 dist-test-slave-sg4j krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:05.490020 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:05.932574 24270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:05.933061 24270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:05.933238 24270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:05.941838 24270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:05.942065 24270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:05.942190 24270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:05.942304 24270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:05.942402 24270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:05.954545 24270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:05.956436 24270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:05.958190 24270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:05.973717 24278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:05.973706 24275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:05.973975 24276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:05.976166 24270 server_base.cc:1034] running on GCE node
I20260927 22:07:05.977510 24270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:05.979758 24270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:05.981235 24270 hybrid_clock.cc:648] HybridClock initialized: now 1790546825981144 us; error 79 us; skew 500 ppm
I20260927 22:07:05.981843 24270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:05 dist-test-slave-sg4j krb5kdc[24257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:05.991978 24270 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:05.994405 24270 webserver.cc:460] Webserver started at http://127.0.0.1:45743/ using document root <none> and password file <none>
I20260927 22:07:05.995465 24270 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:05.995661 24270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:05.996085 24270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:06.000386 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "071471b77bb044cbb5391d24634df768"
format_stamp: "Formatted at 2026-09-27 22:07:05 on dist-test-slave-sg4j"
server_key: "0e343109ae606996587795c0cfa0116d"
server_key_iv: "d942134ca52054b7e551fae75819a5ba"
server_key_version: "encryptionkey@0"
I20260927 22:07:06.001767 24270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "071471b77bb044cbb5391d24634df768"
format_stamp: "Formatted at 2026-09-27 22:07:05 on dist-test-slave-sg4j"
server_key: "0e343109ae606996587795c0cfa0116d"
server_key_iv: "d942134ca52054b7e551fae75819a5ba"
server_key_version: "encryptionkey@0"
I20260927 22:07:06.009775 24270 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 22:07:06.015388 24285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:06.017568 24270 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:07:06.017854 24270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "071471b77bb044cbb5391d24634df768"
format_stamp: "Formatted at 2026-09-27 22:07:05 on dist-test-slave-sg4j"
server_key: "0e343109ae606996587795c0cfa0116d"
server_key_iv: "d942134ca52054b7e551fae75819a5ba"
server_key_version: "encryptionkey@0"
I20260927 22:07:06.018134 24270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:06.049067 24270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:06.054654 24270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:06.055016 24270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:06.079821 24270 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33143
I20260927 22:07:06.080107 24336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33143 every 8 connection(s)
I20260927 22:07:06.082057 24270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:06.087121 24337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:06.087190 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 24270
I20260927 22:07:06.087356 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:06.087949 3708 external_mini_cluster.cc:1428] Setting key 241e1b23844a43bc725dbfeae58a3b47
I20260927 22:07:06.105038 24337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768: Bootstrap starting.
I20260927 22:07:06.108994 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:06.090547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40967 (local address 127.0.0.1:33143)
0927 22:07:06.091778 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:06.091794 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:06.091858 (+ 64us) server_negotiation.cc:407] Connection header received
0927 22:07:06.094986 (+ 3128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:06.095033 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:06.095793 (+ 760us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:06.096300 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:06.097098 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:06.098880 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:06.101064 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:06.102216 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:06.102437 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:06.104734 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:06.104745 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:06.104769 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:06.105733 (+ 964us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:06.105864 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:06.105900 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:06.106068 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:06.107054 (+ 986us) server_negotiation.cc:299] Negotiation successful
0927 22:07:06.107365 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":518,"threads_started":1}
I20260927 22:07:06.111037 24337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:06.112468 24337 log.cc:824] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:06.115094 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:06.117055 24337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768: No bootstrap required, opened a new log
I20260927 22:07:06.129528 24337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071471b77bb044cbb5391d24634df768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260927 22:07:06.130188 24337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:06.130311 24337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 071471b77bb044cbb5391d24634df768, State: Initialized, Role: FOLLOWER
I20260927 22:07:06.130568 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:06.121797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39449 (local address 127.0.0.1:33143)
0927 22:07:06.122108 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:06.122113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:06.122485 (+ 372us) server_negotiation.cc:407] Connection header received
0927 22:07:06.122791 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:06.122798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:06.122907 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:06.123065 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:06.123920 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:06.125020 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:06.127201 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:06.127749 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:06.127876 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:06.129300 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:06.129305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:06.129309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:06.129334 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:06.129468 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:06.129491 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:06.129595 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:06.129836 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 22:07:06.129916 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 22:07:06.131116 24337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071471b77bb044cbb5391d24634df768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260927 22:07:06.131445 24337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:06.131608 24337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:06.131814 24337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:06.134565 24337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071471b77bb044cbb5391d24634df768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260927 22:07:06.135255 24337 leader_election.cc:302] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 071471b77bb044cbb5391d24634df768; no voters:
I20260927 22:07:06.136034 24337 leader_election.cc:288] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:06.136343 24347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:06.138156 24347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [term 1 LEADER]: Becoming Leader. State: Replica: 071471b77bb044cbb5391d24634df768, State: Running, Role: LEADER
I20260927 22:07:06.138980 24337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:06.138986 24347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071471b77bb044cbb5391d24634df768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } }
I20260927 22:07:06.148222 24349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 071471b77bb044cbb5391d24634df768. Latest consensus state: current_term: 1 leader_uuid: "071471b77bb044cbb5391d24634df768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071471b77bb044cbb5391d24634df768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } } }
I20260927 22:07:06.148684 24349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:06.146945 24348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "071471b77bb044cbb5391d24634df768" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "071471b77bb044cbb5391d24634df768" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33143 } } }
I20260927 22:07:06.149319 24348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768 [sys.catalog]: This master's current role is: LEADER
W20260927 22:07:06.153961 24361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:07:06.154349 24361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:07:06.159936 24364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:06.163028 24364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:06.175973 24364 catalog_manager.cc:1347] Generated new cluster ID: 48141080b4bc46438f8f48e63abc09e1
I20260927 22:07:06.176126 24364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:06.192348 24364 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:06.194705 24364 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:06.211226 24364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 071471b77bb044cbb5391d24634df768: Generated new TSK 0
I20260927 22:07:06.212919 24364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:06.231700 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24270
W20260927 22:07:06.233420 24332 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40967
W20260927 22:07:06.234814 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24270: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:06.235100 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 24270
W20260927 22:07:06.256760 24255 connection.cc:535] client connection to 127.0.0.1:33143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33143): Connection reset by peer (error 104)
2026-09-27T22:07:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2255 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:06 dist-test-slave-sg4j krb5kdc[24368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:06 dist-test-slave-sg4j krb5kdc[24368](info): set up 2 sockets
Sep 27 22:07:06 dist-test-slave-sg4j krb5kdc[24368](info): commencing operation
krb5kdc: starting...
W20260927 22:07:07.422435 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:07 dist-test-slave-sg4j krb5kdc[24368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:07.749975 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45007
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:08.186095 24381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:08.186611 24381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:08.187016 24381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:08.197772 24381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:08.197983 24381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:08.198097 24381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:08.198205 24381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:08.198287 24381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:08.215739 24381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45007
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:08.217790 24381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:08.220288 24381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:08.239737 24387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:08.239817 24389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:08.240233 24386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:08.242357 24381 server_base.cc:1034] running on GCE node
I20260927 22:07:08.243974 24381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:08.247453 24381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:08.248929 24381 hybrid_clock.cc:648] HybridClock initialized: now 1790546828248831 us; error 104 us; skew 500 ppm
I20260927 22:07:08.249681 24381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:08 dist-test-slave-sg4j krb5kdc[24368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:08.255767 24381 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:08.259153 24381 webserver.cc:460] Webserver started at http://127.0.0.1:36927/ using document root <none> and password file <none>
I20260927 22:07:08.260334 24381 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:08.260483 24381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:08.260826 24381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:08.265138 24381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "e9b85b356d0e40d4b2d83891501388de"
format_stamp: "Formatted at 2026-09-27 22:07:08 on dist-test-slave-sg4j"
server_key: "a2d2fd14e77b4e985a18bf56e7ea1544"
server_key_iv: "a3489b8d084540165c8a93749b2ac0a2"
server_key_version: "encryptionkey@0"
I20260927 22:07:08.266367 24381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "e9b85b356d0e40d4b2d83891501388de"
format_stamp: "Formatted at 2026-09-27 22:07:08 on dist-test-slave-sg4j"
server_key: "a2d2fd14e77b4e985a18bf56e7ea1544"
server_key_iv: "a3489b8d084540165c8a93749b2ac0a2"
server_key_version: "encryptionkey@0"
I20260927 22:07:08.274255 24381 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 22:07:08.280180 24396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:08.282462 24381 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 22:07:08.282771 24381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "e9b85b356d0e40d4b2d83891501388de"
format_stamp: "Formatted at 2026-09-27 22:07:08 on dist-test-slave-sg4j"
server_key: "a2d2fd14e77b4e985a18bf56e7ea1544"
server_key_iv: "a3489b8d084540165c8a93749b2ac0a2"
server_key_version: "encryptionkey@0"
I20260927 22:07:08.283053 24381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:08.291320 24381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:08.297994 24381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:08.298440 24381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:08.323061 24381 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45007
I20260927 22:07:08.323109 24447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45007 every 8 connection(s)
I20260927 22:07:08.325071 24381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:08.325820 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 24381
I20260927 22:07:08.326061 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:08.326853 3708 external_mini_cluster.cc:1428] Setting key 88f8d73ecd5164b27032957ccdc03f6e
I20260927 22:07:08.331750 24448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:08.338569 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:08.329669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51925 (local address 127.0.0.1:45007)
0927 22:07:08.330832 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:08.330850 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:08.330913 (+ 63us) server_negotiation.cc:407] Connection header received
0927 22:07:08.333137 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:08.333175 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:08.333808 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:08.334289 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:08.335773 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:08.335781 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:08.335800 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:08.335831 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:08.335938 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:08.335968 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:08.336246 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:08.336388 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 22:07:08.336905 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":459,"threads_started":1}
I20260927 22:07:08.344539 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:08.347342 24448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de: Bootstrap starting.
I20260927 22:07:08.351298 24448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:08.353559 24448 log.cc:824] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:08.354861 24451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:08.350198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44867 (local address 127.0.0.1:45007)
0927 22:07:08.350526 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:08.350537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:08.351222 (+ 685us) server_negotiation.cc:407] Connection header received
0927 22:07:08.351405 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:08.351412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:08.351509 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:08.351677 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:08.352851 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:08.352860 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:08.352868 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:08.352898 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:08.352995 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:08.353032 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:08.353481 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:08.353900 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 22:07:08.354006 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 22:07:08.359503 24448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de: No bootstrap required, opened a new log
I20260927 22:07:08.369343 24448 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b85b356d0e40d4b2d83891501388de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45007 } }
I20260927 22:07:08.370007 24448 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:08.370138 24448 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b85b356d0e40d4b2d83891501388de, State: Initialized, Role: FOLLOWER
I20260927 22:07:08.371140 24448 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b85b356d0e40d4b2d83891501388de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45007 } }
I20260927 22:07:08.371462 24448 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:08.371644 24448 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:08.371876 24448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:08.375450 24448 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b85b356d0e40d4b2d83891501388de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45007 } }
I20260927 22:07:08.376381 24448 leader_election.cc:302] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b85b356d0e40d4b2d83891501388de; no voters:
I20260927 22:07:08.377277 24448 leader_election.cc:288] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:08.377653 24458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:08.378796 24458 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [term 1 LEADER]: Becoming Leader. State: Replica: e9b85b356d0e40d4b2d83891501388de, State: Running, Role: LEADER
I20260927 22:07:08.379598 24458 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b85b356d0e40d4b2d83891501388de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45007 } }
I20260927 22:07:08.380739 24448 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:08.386332 24460 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b85b356d0e40d4b2d83891501388de. Latest consensus state: current_term: 1 leader_uuid: "e9b85b356d0e40d4b2d83891501388de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b85b356d0e40d4b2d83891501388de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45007 } } }
I20260927 22:07:08.388746 24459 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b85b356d0e40d4b2d83891501388de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b85b356d0e40d4b2d83891501388de" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45007 } } }
I20260927 22:07:08.389735 24459 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:08.388796 24460 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:08.391877 24468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:08.400096 24468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:08.417162 24468 catalog_manager.cc:1347] Generated new cluster ID: 9ec037324d564698b88cf645a9db7650
I20260927 22:07:08.417367 24468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:08.432720 24468 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:08.433780 24468 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:08.451460 24468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b85b356d0e40d4b2d83891501388de: Generated new TSK 0
I20260927 22:07:08.452914 24468 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:08.473850 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24381
W20260927 22:07:08.475847 24443 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51925
W20260927 22:07:08.477046 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24381: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:08.477289 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 24381
2026-09-27T22:07:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2244 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:08 dist-test-slave-sg4j krb5kdc[24479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:08 dist-test-slave-sg4j krb5kdc[24479](info): set up 2 sockets
Sep 27 22:07:08 dist-test-slave-sg4j krb5kdc[24479](info): commencing operation
krb5kdc: starting...
W20260927 22:07:09.678519 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.004s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:09 dist-test-slave-sg4j krb5kdc[24479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:10.015239 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:10.482470 24492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:10.482921 24492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:10.483034 24492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:10.491555 24492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:10.491747 24492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:10.491802 24492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:10.491865 24492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:10.491907 24492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:10.503975 24492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:10.505919 24492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:10.507730 24492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:10.524072 24500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:10.524135 24497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:10.524116 24498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:10.525175 24492 server_base.cc:1034] running on GCE node
I20260927 22:07:10.526382 24492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:10.528623 24492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:10.529919 24492 hybrid_clock.cc:648] HybridClock initialized: now 1790546830529838 us; error 84 us; skew 500 ppm
I20260927 22:07:10.530426 24492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:10 dist-test-slave-sg4j krb5kdc[24479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:10.534935 24492 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:10.537245 24492 webserver.cc:460] Webserver started at http://127.0.0.1:40021/ using document root <none> and password file <none>
I20260927 22:07:10.538343 24492 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:10.538492 24492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:10.538894 24492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:10.543268 24492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "e90d3cd449234c00a45f6bd34be71755"
format_stamp: "Formatted at 2026-09-27 22:07:10 on dist-test-slave-sg4j"
server_key: "9b5fada479eca519060628cf0c9d9696"
server_key_iv: "4e362e8720a813b87955d315bb93c6f1"
server_key_version: "encryptionkey@0"
I20260927 22:07:10.544932 24492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "e90d3cd449234c00a45f6bd34be71755"
format_stamp: "Formatted at 2026-09-27 22:07:10 on dist-test-slave-sg4j"
server_key: "9b5fada479eca519060628cf0c9d9696"
server_key_iv: "4e362e8720a813b87955d315bb93c6f1"
server_key_version: "encryptionkey@0"
I20260927 22:07:10.553635 24492 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260927 22:07:10.559365 24507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:10.562007 24492 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:07:10.562296 24492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "e90d3cd449234c00a45f6bd34be71755"
format_stamp: "Formatted at 2026-09-27 22:07:10 on dist-test-slave-sg4j"
server_key: "9b5fada479eca519060628cf0c9d9696"
server_key_iv: "4e362e8720a813b87955d315bb93c6f1"
server_key_version: "encryptionkey@0"
I20260927 22:07:10.562619 24492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:10.592087 24492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:10.598042 24492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:10.598376 24492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:10.625931 24492 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40849
I20260927 22:07:10.626014 24558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40849 every 8 connection(s)
I20260927 22:07:10.628044 24492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:10.632854 24559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:10.633983 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 24492
I20260927 22:07:10.634207 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:10.634801 3708 external_mini_cluster.cc:1428] Setting key b175878e53c68f332c2c02e526b7bcbc
I20260927 22:07:10.651155 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755: Bootstrap starting.
Sep 27 22:07:10 dist-test-slave-sg4j krb5kdc[24479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:07:10.657011 24559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:10.658474 24559 log.cc:824] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:10.663532 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755: No bootstrap required, opened a new log
I20260927 22:07:10.669009 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:10.637878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56349 (local address 127.0.0.1:40849)
0927 22:07:10.639079 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:10.639100 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:10.639167 (+ 67us) server_negotiation.cc:407] Connection header received
0927 22:07:10.641238 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:10.641278 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:10.642105 (+ 827us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:10.642648 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:10.644879 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:10.646688 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:10.648843 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:10.649997 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:10.650201 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:10.657097 (+ 6896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:10.657148 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:10.657181 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:10.657270 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:10.661977 (+ 4707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:10.663303 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:10.663315 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:10.663326 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:10.663775 (+ 449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:10.664408 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:10.664413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:10.664415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:10.665299 (+ 884us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:10.665807 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:10.666491 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 22:07:10.667310 (+ 819us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":541,"threads_started":1}
I20260927 22:07:10.674615 24559 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d3cd449234c00a45f6bd34be71755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260927 22:07:10.675060 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:10.675176 24559 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:10.675338 24559 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90d3cd449234c00a45f6bd34be71755, State: Initialized, Role: FOLLOWER
I20260927 22:07:10.676033 24559 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d3cd449234c00a45f6bd34be71755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260927 22:07:10.676306 24559 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:10.676478 24559 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:10.676652 24559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:10.679364 24559 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d3cd449234c00a45f6bd34be71755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260927 22:07:10.680025 24559 leader_election.cc:302] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90d3cd449234c00a45f6bd34be71755; no voters:
I20260927 22:07:10.680814 24559 leader_election.cc:288] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:10.680931 24567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:10.684753 24559 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:10.685886 24567 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [term 1 LEADER]: Becoming Leader. State: Replica: e90d3cd449234c00a45f6bd34be71755, State: Running, Role: LEADER
I20260927 22:07:10.687006 24567 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d3cd449234c00a45f6bd34be71755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } }
I20260927 22:07:10.692646 24569 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90d3cd449234c00a45f6bd34be71755. Latest consensus state: current_term: 1 leader_uuid: "e90d3cd449234c00a45f6bd34be71755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d3cd449234c00a45f6bd34be71755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } } }
I20260927 22:07:10.693765 24568 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e90d3cd449234c00a45f6bd34be71755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90d3cd449234c00a45f6bd34be71755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40849 } } }
I20260927 22:07:10.693326 24569 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:10.694100 24568 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:10.700232 24576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:10.712634 24576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:10.718101 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:10.689704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37411 (local address 127.0.0.1:40849)
0927 22:07:10.690170 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:10.690178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:10.690529 (+ 351us) server_negotiation.cc:407] Connection header received
0927 22:07:10.691085 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:10.691092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:10.691200 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:10.691307 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:10.694834 (+ 3527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:10.696947 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:10.699880 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:10.701073 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:10.701260 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:10.706588 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:10.706596 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:10.706601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:10.706660 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:10.712065 (+ 5405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:10.713750 (+ 1685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:10.713757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:10.713761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:10.714000 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:10.714617 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:10.714627 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:10.714631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:10.715202 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:10.717024 (+ 1822us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:10.717232 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 22:07:10.717620 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274}
I20260927 22:07:10.730976 24576 catalog_manager.cc:1347] Generated new cluster ID: 3810108b07c3457eb34a3d569d1b2faa
I20260927 22:07:10.731149 24576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:10.748558 24576 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:10.750998 24576 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:10.767467 24576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90d3cd449234c00a45f6bd34be71755: Generated new TSK 0
I20260927 22:07:10.768612 24576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:10.779232 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24492
I20260927 22:07:10.780646 24554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:10.883019 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 24492
2026-09-27T22:07:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:11 dist-test-slave-sg4j krb5kdc[24592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:11 dist-test-slave-sg4j krb5kdc[24592](info): set up 2 sockets
Sep 27 22:07:11 dist-test-slave-sg4j krb5kdc[24592](info): commencing operation
krb5kdc: starting...
W20260927 22:07:12.066967 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.138s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:12 dist-test-slave-sg4j krb5kdc[24592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:12.408118 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:12.846949 24605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:12.847453 24605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:12.847642 24605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:12.856724 24605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:12.856953 24605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:12.857074 24605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:12.857191 24605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:12.857275 24605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:12.869306 24605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:12.871312 24605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:12.873204 24605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:12.889513 24613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:12.889598 24610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:12.889575 24611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:12.890754 24605 server_base.cc:1034] running on GCE node
I20260927 22:07:12.892200 24605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:12.894340 24605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:12.895763 24605 hybrid_clock.cc:648] HybridClock initialized: now 1790546832895670 us; error 88 us; skew 500 ppm
I20260927 22:07:12.896288 24605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:12 dist-test-slave-sg4j krb5kdc[24592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:12.906059 24605 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:12.909504 24605 webserver.cc:460] Webserver started at http://127.0.0.1:43853/ using document root <none> and password file <none>
I20260927 22:07:12.910804 24605 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:12.911012 24605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:12.911561 24605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:12.917699 24605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "e75778f61feb4e5898a475c669565970"
format_stamp: "Formatted at 2026-09-27 22:07:12 on dist-test-slave-sg4j"
server_key: "d057ac9ff11c1ed1131a3502e43ee2e5"
server_key_iv: "049048b6b2ca919ecf1989ce56a57345"
server_key_version: "encryptionkey@0"
I20260927 22:07:12.919001 24605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "e75778f61feb4e5898a475c669565970"
format_stamp: "Formatted at 2026-09-27 22:07:12 on dist-test-slave-sg4j"
server_key: "d057ac9ff11c1ed1131a3502e43ee2e5"
server_key_iv: "049048b6b2ca919ecf1989ce56a57345"
server_key_version: "encryptionkey@0"
I20260927 22:07:12.926631 24605 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260927 22:07:12.932224 24620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:12.934974 24605 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s
I20260927 22:07:12.935292 24605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "e75778f61feb4e5898a475c669565970"
format_stamp: "Formatted at 2026-09-27 22:07:12 on dist-test-slave-sg4j"
server_key: "d057ac9ff11c1ed1131a3502e43ee2e5"
server_key_iv: "049048b6b2ca919ecf1989ce56a57345"
server_key_version: "encryptionkey@0"
I20260927 22:07:12.935796 24605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:12.968782 24605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:12.974442 24605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:12.974839 24605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:12.998881 24605 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40917
I20260927 22:07:12.998929 24671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40917 every 8 connection(s)
I20260927 22:07:13.000538 24605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:13.005988 24672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:13.006815 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 24605
I20260927 22:07:13.007004 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:13.007539 3708 external_mini_cluster.cc:1428] Setting key fa7d86b5db3634fb39301f28ce14c8cf
I20260927 22:07:13.024912 24672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970: Bootstrap starting.
I20260927 22:07:13.027377 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:13.010774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60899 (local address 127.0.0.1:40917)
0927 22:07:13.011830 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:13.011845 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:13.011893 (+ 48us) server_negotiation.cc:407] Connection header received
0927 22:07:13.014634 (+ 2741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:13.014674 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:13.015266 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:13.015727 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:13.016710 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:13.018259 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:13.020712 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:13.021858 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:13.022081 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:13.023857 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:13.023868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:13.023889 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:13.024859 (+ 970us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:13.025007 (+ 148us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:13.025046 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:13.025255 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:13.025593 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 22:07:13.026035 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":485,"threads_started":1}
I20260927 22:07:13.029789 24672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:13.031716 24672 log.cc:824] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:13.033424 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:13.036060 24672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970: No bootstrap required, opened a new log
I20260927 22:07:13.047664 24672 raft_consensus.cc:348] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75778f61feb4e5898a475c669565970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260927 22:07:13.048256 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:13.039595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37219 (local address 127.0.0.1:40917)
0927 22:07:13.039876 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:13.039883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:13.040298 (+ 415us) server_negotiation.cc:407] Connection header received
0927 22:07:13.040587 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:13.040596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:13.040746 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:13.040974 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:13.041665 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:13.042609 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:13.044782 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:13.045559 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:13.045691 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:13.047303 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:13.047308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:13.047312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:13.047338 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:13.047388 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:13.047464 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:13.047551 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:13.047745 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 22:07:13.047845 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260927 22:07:13.048344 24672 raft_consensus.cc:374] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:13.048617 24672 raft_consensus.cc:729] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75778f61feb4e5898a475c669565970, State: Initialized, Role: FOLLOWER
I20260927 22:07:13.049629 24672 consensus_queue.cc:260] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75778f61feb4e5898a475c669565970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260927 22:07:13.050019 24672 raft_consensus.cc:388] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:13.050261 24672 raft_consensus.cc:482] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:13.050930 24672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:13.053807 24672 raft_consensus.cc:504] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75778f61feb4e5898a475c669565970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260927 22:07:13.054443 24672 leader_election.cc:302] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e75778f61feb4e5898a475c669565970; no voters:
I20260927 22:07:13.055151 24672 leader_election.cc:288] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:13.055469 24682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:13.057149 24682 raft_consensus.cc:686] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [term 1 LEADER]: Becoming Leader. State: Replica: e75778f61feb4e5898a475c669565970, State: Running, Role: LEADER
I20260927 22:07:13.057934 24682 consensus_queue.cc:237] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75778f61feb4e5898a475c669565970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260927 22:07:13.058547 24672 sys_catalog.cc:564] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:13.061619 24684 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e75778f61feb4e5898a475c669565970. Latest consensus state: current_term: 1 leader_uuid: "e75778f61feb4e5898a475c669565970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75778f61feb4e5898a475c669565970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } } }
I20260927 22:07:13.062083 24683 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e75778f61feb4e5898a475c669565970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75778f61feb4e5898a475c669565970" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } } }
I20260927 22:07:13.062513 24684 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:13.062891 24683 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:13.063886 24688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:13.070744 24688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:13.084553 24688 catalog_manager.cc:1347] Generated new cluster ID: d41ddd5718b94782a8207879b8e2a9ca
I20260927 22:07:13.084740 24688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:13.096968 24688 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:13.098692 24688 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:13.107962 24688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e75778f61feb4e5898a475c669565970: Generated new TSK 0
I20260927 22:07:13.109699 24688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:13.128657 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24605
W20260927 22:07:13.130517 24667 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60899
W20260927 22:07:13.131523 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24605: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:13.131745 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 24605
2026-09-27T22:07:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:13 dist-test-slave-sg4j krb5kdc[24703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:13 dist-test-slave-sg4j krb5kdc[24703](info): set up 2 sockets
Sep 27 22:07:13 dist-test-slave-sg4j krb5kdc[24703](info): commencing operation
krb5kdc: starting...
W20260927 22:07:14.286527 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:14 dist-test-slave-sg4j krb5kdc[24703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:14.634598 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:15.078137 24716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:15.078631 24716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:15.078830 24716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:15.090392 24716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:15.090772 24716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:15.090919 24716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:15.091053 24716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:15.091144 24716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:15.104988 24716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:15.107234 24716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:15.110626 24716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:15.131179 24722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:15.131147 24724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:15.131531 24721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:15.132472 24716 server_base.cc:1034] running on GCE node
I20260927 22:07:15.133694 24716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:15.135938 24716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:15.137297 24716 hybrid_clock.cc:648] HybridClock initialized: now 1790546835137234 us; error 61 us; skew 500 ppm
I20260927 22:07:15.137928 24716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:15 dist-test-slave-sg4j krb5kdc[24703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:15.142792 24716 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:15.145185 24716 webserver.cc:460] Webserver started at http://127.0.0.1:46451/ using document root <none> and password file <none>
I20260927 22:07:15.146327 24716 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:15.146534 24716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:15.146945 24716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:15.151278 24716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "376a6323f6614ad1893820aac775d0ea"
format_stamp: "Formatted at 2026-09-27 22:07:15 on dist-test-slave-sg4j"
server_key: "c92da942693a0eb47d01d20a51e41397"
server_key_iv: "4ffb3ea55313c7165746dff8e3799f87"
server_key_version: "encryptionkey@0"
I20260927 22:07:15.152421 24716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "376a6323f6614ad1893820aac775d0ea"
format_stamp: "Formatted at 2026-09-27 22:07:15 on dist-test-slave-sg4j"
server_key: "c92da942693a0eb47d01d20a51e41397"
server_key_iv: "4ffb3ea55313c7165746dff8e3799f87"
server_key_version: "encryptionkey@0"
I20260927 22:07:15.160315 24716 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260927 22:07:15.166431 24731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:15.168850 24716 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:07:15.169219 24716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "376a6323f6614ad1893820aac775d0ea"
format_stamp: "Formatted at 2026-09-27 22:07:15 on dist-test-slave-sg4j"
server_key: "c92da942693a0eb47d01d20a51e41397"
server_key_iv: "4ffb3ea55313c7165746dff8e3799f87"
server_key_version: "encryptionkey@0"
I20260927 22:07:15.169577 24716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:15.179169 24716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:15.186589 24716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:15.187052 24716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:15.212880 24716 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38205
I20260927 22:07:15.212958 24782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38205 every 8 connection(s)
I20260927 22:07:15.214882 24716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:15.220680 24783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:15.221220 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 24716
I20260927 22:07:15.221487 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:15.222137 3708 external_mini_cluster.cc:1428] Setting key e30783684310249e572bf8207bce39bd
I20260927 22:07:15.233443 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:15.224705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34927 (local address 127.0.0.1:38205)
0927 22:07:15.225977 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:15.225995 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:15.226062 (+ 67us) server_negotiation.cc:407] Connection header received
0927 22:07:15.228364 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:15.228406 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:15.229011 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:15.229627 (+ 616us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:15.231136 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:15.231144 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:15.231163 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:15.231182 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:15.231273 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:15.231301 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:15.231597 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:15.231727 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 22:07:15.232064 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":609,"threads_started":1}
I20260927 22:07:15.237920 24783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea: Bootstrap starting.
I20260927 22:07:15.238996 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:15.243086 24783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:15.244989 24783 log.cc:824] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:15.249138 24783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea: No bootstrap required, opened a new log
I20260927 22:07:15.249547 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:15.246379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38791 (local address 127.0.0.1:38205)
0927 22:07:15.246647 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:15.246655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:15.246946 (+ 291us) server_negotiation.cc:407] Connection header received
0927 22:07:15.247191 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:15.247196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:15.247265 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:15.247393 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:15.248543 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:15.248549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:15.248553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:15.248574 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:15.248629 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:15.248648 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:15.248797 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:15.248885 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 22:07:15.248975 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260927 22:07:15.258973 24783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376a6323f6614ad1893820aac775d0ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } }
I20260927 22:07:15.259481 24783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:15.259562 24783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376a6323f6614ad1893820aac775d0ea, State: Initialized, Role: FOLLOWER
I20260927 22:07:15.260419 24783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376a6323f6614ad1893820aac775d0ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } }
I20260927 22:07:15.260720 24783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:15.260951 24783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:15.261163 24783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:15.265196 24783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376a6323f6614ad1893820aac775d0ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } }
I20260927 22:07:15.266423 24783 leader_election.cc:302] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 376a6323f6614ad1893820aac775d0ea; no voters:
I20260927 22:07:15.267475 24783 leader_election.cc:288] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:15.267721 24793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:15.269184 24793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [term 1 LEADER]: Becoming Leader. State: Replica: 376a6323f6614ad1893820aac775d0ea, State: Running, Role: LEADER
I20260927 22:07:15.270550 24793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376a6323f6614ad1893820aac775d0ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } }
I20260927 22:07:15.272818 24783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:15.275871 24795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 376a6323f6614ad1893820aac775d0ea. Latest consensus state: current_term: 1 leader_uuid: "376a6323f6614ad1893820aac775d0ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376a6323f6614ad1893820aac775d0ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } } }
I20260927 22:07:15.276021 24794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "376a6323f6614ad1893820aac775d0ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "376a6323f6614ad1893820aac775d0ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } } }
I20260927 22:07:15.278438 24795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:15.278438 24794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:15.282226 24799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:15.290958 24799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:15.304190 24799 catalog_manager.cc:1347] Generated new cluster ID: 06980eaad74d4fd6abeda0b93fd79039
I20260927 22:07:15.304374 24799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:15.323411 24799 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:15.323994 24799 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:15.335230 24799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 376a6323f6614ad1893820aac775d0ea: Generated new TSK 0
I20260927 22:07:15.336447 24799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:15.347627 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24716
W20260927 22:07:15.349731 24778 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34927
W20260927 22:07:15.350836 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24716: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:15.351090 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 24716
2026-09-27T22:07:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:15 dist-test-slave-sg4j krb5kdc[24814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:15 dist-test-slave-sg4j krb5kdc[24814](info): set up 2 sockets
Sep 27 22:07:15 dist-test-slave-sg4j krb5kdc[24814](info): commencing operation
krb5kdc: starting...
W20260927 22:07:16.530778 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.002s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:16 dist-test-slave-sg4j krb5kdc[24814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:16.882999 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:17.317500 24827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:17.318001 24827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:17.318253 24827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:17.327131 24827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:17.327310 24827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:17.327404 24827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:17.327502 24827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:17.327545 24827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:17.339496 24827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:17.341346 24827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:17.343118 24827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:17.359551 24832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:17.359740 24833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:17.359712 24835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:17.361627 24827 server_base.cc:1034] running on GCE node
I20260927 22:07:17.362819 24827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:17.364794 24827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:17.366307 24827 hybrid_clock.cc:648] HybridClock initialized: now 1790546837366191 us; error 90 us; skew 500 ppm
I20260927 22:07:17.366881 24827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:17 dist-test-slave-sg4j krb5kdc[24814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:17.371140 24827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:17.373574 24827 webserver.cc:460] Webserver started at http://127.0.0.1:42609/ using document root <none> and password file <none>
I20260927 22:07:17.374481 24827 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:17.374655 24827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:17.375023 24827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:17.379406 24827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "d3076002b9bb409d8c88d500325310f7"
format_stamp: "Formatted at 2026-09-27 22:07:17 on dist-test-slave-sg4j"
server_key: "49cfabff3fe774657cf98472153e518a"
server_key_iv: "55dd12af0ea6cde34fa355462510572d"
server_key_version: "encryptionkey@0"
I20260927 22:07:17.380507 24827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "d3076002b9bb409d8c88d500325310f7"
format_stamp: "Formatted at 2026-09-27 22:07:17 on dist-test-slave-sg4j"
server_key: "49cfabff3fe774657cf98472153e518a"
server_key_iv: "55dd12af0ea6cde34fa355462510572d"
server_key_version: "encryptionkey@0"
I20260927 22:07:17.388096 24827 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260927 22:07:17.394155 24842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:17.396826 24827 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260927 22:07:17.397136 24827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "d3076002b9bb409d8c88d500325310f7"
format_stamp: "Formatted at 2026-09-27 22:07:17 on dist-test-slave-sg4j"
server_key: "49cfabff3fe774657cf98472153e518a"
server_key_iv: "55dd12af0ea6cde34fa355462510572d"
server_key_version: "encryptionkey@0"
I20260927 22:07:17.397475 24827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:17.420077 24827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:17.425343 24827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:17.425727 24827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:17.450548 24827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42553
I20260927 22:07:17.450572 24893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42553 every 8 connection(s)
I20260927 22:07:17.452201 24827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:17.457568 24894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:17.458655 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 24827
I20260927 22:07:17.458907 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:17.459419 3708 external_mini_cluster.cc:1428] Setting key 63e581d515cd5e4f56d3ae583f147ba0
I20260927 22:07:17.474789 24894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7: Bootstrap starting.
Sep 27 22:07:17 dist-test-slave-sg4j krb5kdc[24814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:07:17.479918 24894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:17.481923 24894 log.cc:824] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:17.486852 24894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7: No bootstrap required, opened a new log
I20260927 22:07:17.490314 24897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:17.461818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:44897 (local address 127.0.0.1:42553)
0927 22:07:17.462962 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:17.462983 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:17.463041 (+ 58us) server_negotiation.cc:407] Connection header received
0927 22:07:17.465880 (+ 2839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:17.465915 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:17.466522 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:17.467015 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:17.468639 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:17.470207 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:17.471699 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:17.472516 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:17.472762 (+ 246us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:17.478620 (+ 5858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:17.478639 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:17.478668 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:17.478747 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:17.483997 (+ 5250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:17.485242 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:17.485253 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:17.485264 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:17.485571 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:17.486163 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:17.486172 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:17.486175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:17.487079 (+ 904us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:17.487506 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:17.488055 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 22:07:17.488515 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":496,"threads_started":1}
I20260927 22:07:17.496187 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:17.498701 24894 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3076002b9bb409d8c88d500325310f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42553 } }
I20260927 22:07:17.499264 24894 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:17.499447 24894 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3076002b9bb409d8c88d500325310f7, State: Initialized, Role: FOLLOWER
I20260927 22:07:17.500310 24894 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3076002b9bb409d8c88d500325310f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42553 } }
I20260927 22:07:17.500646 24894 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:17.500844 24894 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:17.501057 24894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:17.504782 24894 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3076002b9bb409d8c88d500325310f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42553 } }
I20260927 22:07:17.505571 24894 leader_election.cc:302] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3076002b9bb409d8c88d500325310f7; no voters:
I20260927 22:07:17.506443 24894 leader_election.cc:288] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:17.506567 24904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:17.508351 24904 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [term 1 LEADER]: Becoming Leader. State: Replica: d3076002b9bb409d8c88d500325310f7, State: Running, Role: LEADER
I20260927 22:07:17.509259 24904 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3076002b9bb409d8c88d500325310f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42553 } }
I20260927 22:07:17.509716 24894 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:17.519775 24897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:17.501662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:53933 (local address 127.0.0.1:42553)
0927 22:07:17.501936 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:17.501949 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:17.502422 (+ 473us) server_negotiation.cc:407] Connection header received
0927 22:07:17.502666 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:17.502670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:17.502747 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:17.502878 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:17.504925 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:17.505981 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:17.508074 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:17.508825 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:17.508980 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:17.512173 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:17.512190 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:17.512195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:17.512255 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:17.515669 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:17.516781 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:17.516785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:17.516788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:17.516917 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:17.517446 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:17.517454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:17.517456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:17.517816 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:17.518145 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:17.519049 (+ 904us) server_negotiation.cc:299] Negotiation successful
0927 22:07:17.519260 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 22:07:17.520009 24905 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3076002b9bb409d8c88d500325310f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3076002b9bb409d8c88d500325310f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42553 } } }
I20260927 22:07:17.520609 24905 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:17.522568 24906 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3076002b9bb409d8c88d500325310f7. Latest consensus state: current_term: 1 leader_uuid: "d3076002b9bb409d8c88d500325310f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3076002b9bb409d8c88d500325310f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42553 } } }
I20260927 22:07:17.523049 24906 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:17.524268 24914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:17.531450 24914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:17.544978 24914 catalog_manager.cc:1347] Generated new cluster ID: e959d71b556c408eab03d1bd591a1944
I20260927 22:07:17.545150 24914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:17.556820 24914 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:17.558652 24914 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:17.567453 24914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3076002b9bb409d8c88d500325310f7: Generated new TSK 0
I20260927 22:07:17.569169 24914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:17.585757 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24827
I20260927 22:07:17.587503 24889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:17.697711 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 24827
W20260927 22:07:17.717278 24812 connection.cc:535] client connection to 127.0.0.1:42553 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42553): Connection reset by peer (error 104)
2026-09-27T22:07:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:17 dist-test-slave-sg4j krb5kdc[24927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:17 dist-test-slave-sg4j krb5kdc[24927](info): set up 2 sockets
Sep 27 22:07:17 dist-test-slave-sg4j krb5kdc[24927](info): commencing operation
krb5kdc: starting...
W20260927 22:07:18.895277 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.010s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:19 dist-test-slave-sg4j krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:19.246359 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:19.711249 24940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:19.712076 24940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:19.712289 24940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:19.721246 24940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:19.721678 24940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:19.721828 24940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:19.721961 24940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:19.722054 24940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:19.734342 24940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:19.735878 24940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:19.737675 24940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:19.752499 24945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:19.752774 24946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:19.753029 24948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:19.754817 24940 server_base.cc:1034] running on GCE node
I20260927 22:07:19.756035 24940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:19.759042 24940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:19.760720 24940 hybrid_clock.cc:648] HybridClock initialized: now 1790546839760584 us; error 130 us; skew 500 ppm
I20260927 22:07:19.761269 24940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:19 dist-test-slave-sg4j krb5kdc[24927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:19.765821 24940 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:19.768539 24940 webserver.cc:460] Webserver started at http://127.0.0.1:38625/ using document root <none> and password file <none>
I20260927 22:07:19.769550 24940 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:19.769737 24940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:19.770093 24940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:19.773934 24940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "1ef8b3665dc0493784b1b9a2b8f7f4ea"
format_stamp: "Formatted at 2026-09-27 22:07:19 on dist-test-slave-sg4j"
server_key: "c3726112c84e290e1f82f6a62cf80166"
server_key_iv: "acc184c262e0a2cee008b43f05c82858"
server_key_version: "encryptionkey@0"
I20260927 22:07:19.774892 24940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "1ef8b3665dc0493784b1b9a2b8f7f4ea"
format_stamp: "Formatted at 2026-09-27 22:07:19 on dist-test-slave-sg4j"
server_key: "c3726112c84e290e1f82f6a62cf80166"
server_key_iv: "acc184c262e0a2cee008b43f05c82858"
server_key_version: "encryptionkey@0"
I20260927 22:07:19.782505 24940 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:07:19.787395 24955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:19.790038 24940 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260927 22:07:19.790362 24940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "1ef8b3665dc0493784b1b9a2b8f7f4ea"
format_stamp: "Formatted at 2026-09-27 22:07:19 on dist-test-slave-sg4j"
server_key: "c3726112c84e290e1f82f6a62cf80166"
server_key_iv: "acc184c262e0a2cee008b43f05c82858"
server_key_version: "encryptionkey@0"
I20260927 22:07:19.790958 24940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:19.814196 24940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:19.819366 24940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:19.819736 24940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:19.845779 24940 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46575
I20260927 22:07:19.845849 25006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46575 every 8 connection(s)
I20260927 22:07:19.847538 24940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:19.853284 25007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:19.856355 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 24940
I20260927 22:07:19.856537 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:19.857071 3708 external_mini_cluster.cc:1428] Setting key e9584b38e264032435a8dc8c06d22b4c
I20260927 22:07:19.871168 25007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea: Bootstrap starting.
I20260927 22:07:19.876730 25007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:19.877753 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:19.859748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:35451 (local address 127.0.0.1:46575)
0927 22:07:19.860914 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:19.860927 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:19.860986 (+ 59us) server_negotiation.cc:407] Connection header received
0927 22:07:19.863815 (+ 2829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:19.863850 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:19.864463 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:19.864895 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:19.866058 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:19.867730 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:19.870116 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:19.871450 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:19.871844 (+ 394us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:19.873821 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:19.873831 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:19.873855 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:19.874591 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:19.874701 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:19.874740 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:19.874921 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:19.875503 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 22:07:19.875752 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":523,"threads_started":1}
I20260927 22:07:19.878798 25007 log.cc:824] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:19.883884 25007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea: No bootstrap required, opened a new log
I20260927 22:07:19.885154 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:19.896744 25007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef8b3665dc0493784b1b9a2b8f7f4ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } }
I20260927 22:07:19.897446 25007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:19.897565 25007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef8b3665dc0493784b1b9a2b8f7f4ea, State: Initialized, Role: FOLLOWER
I20260927 22:07:19.898530 25007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef8b3665dc0493784b1b9a2b8f7f4ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } }
I20260927 22:07:19.898913 25007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:19.899104 25007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:19.899358 25007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:19.902145 25010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:19.892390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:49879 (local address 127.0.0.1:46575)
0927 22:07:19.892706 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:19.892715 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:19.893701 (+ 986us) server_negotiation.cc:407] Connection header received
0927 22:07:19.893997 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:19.894004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:19.894163 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:19.894340 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:19.895148 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:19.896543 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:19.898591 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:19.899395 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:19.899598 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:19.901017 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:19.901027 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:19.901033 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:19.901064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:19.901162 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:19.901192 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:19.901370 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:19.901690 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 22:07:19.901772 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 22:07:19.903028 25007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef8b3665dc0493784b1b9a2b8f7f4ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } }
I20260927 22:07:19.903800 25007 leader_election.cc:302] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ef8b3665dc0493784b1b9a2b8f7f4ea; no voters:
I20260927 22:07:19.904595 25007 leader_election.cc:288] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:19.904887 25017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:19.905848 25017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [term 1 LEADER]: Becoming Leader. State: Replica: 1ef8b3665dc0493784b1b9a2b8f7f4ea, State: Running, Role: LEADER
I20260927 22:07:19.906598 25017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef8b3665dc0493784b1b9a2b8f7f4ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } }
I20260927 22:07:19.907428 25007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:19.910872 25018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ef8b3665dc0493784b1b9a2b8f7f4ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef8b3665dc0493784b1b9a2b8f7f4ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } } }
I20260927 22:07:19.911432 25018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:19.912549 25019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ef8b3665dc0493784b1b9a2b8f7f4ea. Latest consensus state: current_term: 1 leader_uuid: "1ef8b3665dc0493784b1b9a2b8f7f4ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ef8b3665dc0493784b1b9a2b8f7f4ea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46575 } } }
I20260927 22:07:19.913172 25019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:19.915735 25025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:19.924944 25025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:19.939623 25025 catalog_manager.cc:1347] Generated new cluster ID: 64e03fd17aa74ff992909716be595001
I20260927 22:07:19.939841 25025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:19.962976 25025 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:19.964949 25025 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:19.981976 25025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ef8b3665dc0493784b1b9a2b8f7f4ea: Generated new TSK 0
I20260927 22:07:19.983680 25025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:20.001226 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24940
W20260927 22:07:20.003216 25002 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.143:35451
W20260927 22:07:20.004688 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 24940: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:20.004877 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 24940
W20260927 22:07:20.026903 24925 connection.cc:535] client connection to 127.0.0.1:46575 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46575): Connection reset by peer (error 104)
2026-09-27T22:07:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:20 dist-test-slave-sg4j krb5kdc[25038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:20 dist-test-slave-sg4j krb5kdc[25038](info): set up 2 sockets
Sep 27 22:07:20 dist-test-slave-sg4j krb5kdc[25038](info): commencing operation
krb5kdc: starting...
W20260927 22:07:21.218743 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:21 dist-test-slave-sg4j krb5kdc[25038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:21.570693 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:22.015080 25051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:22.015588 25051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:22.015763 25051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:22.024765 25051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:22.024997 25051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:22.025122 25051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:22.025229 25051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:22.025319 25051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:22.036949 25051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:22.038853 25051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:22.040575 25051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:22.056874 25056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:22.056879 25057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:22.056879 25059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:22.058336 25051 server_base.cc:1034] running on GCE node
I20260927 22:07:22.059455 25051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:22.061553 25051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:22.062810 25051 hybrid_clock.cc:648] HybridClock initialized: now 1790546842062724 us; error 78 us; skew 500 ppm
I20260927 22:07:22.063336 25051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:22 dist-test-slave-sg4j krb5kdc[25038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:22.068179 25051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:22.070727 25051 webserver.cc:460] Webserver started at http://127.0.0.1:46089/ using document root <none> and password file <none>
I20260927 22:07:22.071879 25051 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:22.072108 25051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:22.072536 25051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:22.076932 25051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "62f68dfffcec4cb5bf4732ac2e9aca7f"
format_stamp: "Formatted at 2026-09-27 22:07:22 on dist-test-slave-sg4j"
server_key: "6ee17f94993b4a5fe10f890f508bea78"
server_key_iv: "d3c326f63c9483091eda879cdec0dae4"
server_key_version: "encryptionkey@0"
I20260927 22:07:22.078166 25051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "62f68dfffcec4cb5bf4732ac2e9aca7f"
format_stamp: "Formatted at 2026-09-27 22:07:22 on dist-test-slave-sg4j"
server_key: "6ee17f94993b4a5fe10f890f508bea78"
server_key_iv: "d3c326f63c9483091eda879cdec0dae4"
server_key_version: "encryptionkey@0"
I20260927 22:07:22.086251 25051 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260927 22:07:22.092089 25066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:22.094854 25051 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 22:07:22.095206 25051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "62f68dfffcec4cb5bf4732ac2e9aca7f"
format_stamp: "Formatted at 2026-09-27 22:07:22 on dist-test-slave-sg4j"
server_key: "6ee17f94993b4a5fe10f890f508bea78"
server_key_iv: "d3c326f63c9483091eda879cdec0dae4"
server_key_version: "encryptionkey@0"
I20260927 22:07:22.095643 25051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:22.103804 25051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:22.110448 25051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:22.110898 25051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:22.137863 25051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46031
I20260927 22:07:22.137967 25117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46031 every 8 connection(s)
I20260927 22:07:22.139714 25051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:22.140272 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 25051
I20260927 22:07:22.140623 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:22.141467 3708 external_mini_cluster.cc:1428] Setting key 44cb55beb3116075cb25a3257aa1c052
I20260927 22:07:22.145888 25118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:22.152738 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:22.144098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:55209 (local address 127.0.0.1:46031)
0927 22:07:22.145146 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:22.145161 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:22.145220 (+ 59us) server_negotiation.cc:407] Connection header received
0927 22:07:22.147324 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:22.147355 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:22.148027 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:22.148862 (+ 835us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:22.150442 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:22.150452 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:22.150474 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:22.150536 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:22.150671 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:22.150729 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:22.151002 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:22.151179 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 22:07:22.151499 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":487,"threads_started":1}
I20260927 22:07:22.159253 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:22.163259 25118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f: Bootstrap starting.
I20260927 22:07:22.169309 25121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:22.165713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:37589 (local address 127.0.0.1:46031)
0927 22:07:22.165971 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:22.165976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:22.166613 (+ 637us) server_negotiation.cc:407] Connection header received
0927 22:07:22.166806 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:22.166813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:22.166905 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:22.167156 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:22.168354 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:22.168360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:22.168365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:22.168398 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:22.168472 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:22.168495 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:22.168652 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:22.168823 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 22:07:22.168911 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260927 22:07:22.169271 25118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:22.171123 25118 log.cc:824] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:22.176000 25118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f: No bootstrap required, opened a new log
I20260927 22:07:22.187460 25118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f68dfffcec4cb5bf4732ac2e9aca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46031 } }
I20260927 22:07:22.188225 25118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:22.188361 25118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62f68dfffcec4cb5bf4732ac2e9aca7f, State: Initialized, Role: FOLLOWER
I20260927 22:07:22.189206 25118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f68dfffcec4cb5bf4732ac2e9aca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46031 } }
I20260927 22:07:22.189590 25118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:22.189757 25118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:22.189999 25118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:22.193327 25118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f68dfffcec4cb5bf4732ac2e9aca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46031 } }
I20260927 22:07:22.194440 25118 leader_election.cc:302] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62f68dfffcec4cb5bf4732ac2e9aca7f; no voters:
I20260927 22:07:22.195626 25118 leader_election.cc:288] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:22.195807 25128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:22.197147 25128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [term 1 LEADER]: Becoming Leader. State: Replica: 62f68dfffcec4cb5bf4732ac2e9aca7f, State: Running, Role: LEADER
I20260927 22:07:22.198294 25128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f68dfffcec4cb5bf4732ac2e9aca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46031 } }
I20260927 22:07:22.199141 25118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:22.201931 25129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62f68dfffcec4cb5bf4732ac2e9aca7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f68dfffcec4cb5bf4732ac2e9aca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46031 } } }
I20260927 22:07:22.202391 25129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:22.203395 25133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:22.206105 25130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62f68dfffcec4cb5bf4732ac2e9aca7f. Latest consensus state: current_term: 1 leader_uuid: "62f68dfffcec4cb5bf4732ac2e9aca7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62f68dfffcec4cb5bf4732ac2e9aca7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46031 } } }
I20260927 22:07:22.206835 25130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:22.216401 25133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:22.234771 25133 catalog_manager.cc:1347] Generated new cluster ID: 207ed094ebc14aadb71b393d3dc73fe6
I20260927 22:07:22.234977 25133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:22.274520 25133 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:22.275216 25133 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:22.292217 25133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62f68dfffcec4cb5bf4732ac2e9aca7f: Generated new TSK 0
I20260927 22:07:22.293993 25133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:22.304397 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25051
W20260927 22:07:22.305979 25113 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.143:55209
W20260927 22:07:22.306969 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25051: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:22.307161 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 25051
2026-09-27T22:07:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:22 dist-test-slave-sg4j krb5kdc[25149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:22 dist-test-slave-sg4j krb5kdc[25149](info): set up 2 sockets
Sep 27 22:07:22 dist-test-slave-sg4j krb5kdc[25149](info): commencing operation
krb5kdc: starting...
W20260927 22:07:23.538761 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:23 dist-test-slave-sg4j krb5kdc[25149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:23.890695 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:24.369232 25162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:24.370059 25162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:24.370304 25162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:24.381165 25162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:24.381472 25162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:24.381574 25162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:24.381701 25162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:24.381752 25162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:24.396247 25162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:24.399098 25162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:24.402010 25162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:24.421582 25168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:24.422287 25170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:24.422465 25167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:24.423956 25162 server_base.cc:1034] running on GCE node
I20260927 22:07:24.425334 25162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:24.428347 25162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:24.430084 25162 hybrid_clock.cc:648] HybridClock initialized: now 1790546844429993 us; error 83 us; skew 500 ppm
I20260927 22:07:24.430651 25162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:24 dist-test-slave-sg4j krb5kdc[25149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:24.435877 25162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:24.439388 25162 webserver.cc:460] Webserver started at http://127.0.0.1:44591/ using document root <none> and password file <none>
I20260927 22:07:24.440630 25162 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:24.440882 25162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:24.441825 25162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:24.447345 25162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "4a1d6abb3e1543a9bb83a63fc1bec90f"
format_stamp: "Formatted at 2026-09-27 22:07:24 on dist-test-slave-sg4j"
server_key: "4b3571068dbb02edefa995fa59dfe454"
server_key_iv: "dfc4d3dbf80583db0da31e2d52b6ff52"
server_key_version: "encryptionkey@0"
I20260927 22:07:24.449249 25162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "4a1d6abb3e1543a9bb83a63fc1bec90f"
format_stamp: "Formatted at 2026-09-27 22:07:24 on dist-test-slave-sg4j"
server_key: "4b3571068dbb02edefa995fa59dfe454"
server_key_iv: "dfc4d3dbf80583db0da31e2d52b6ff52"
server_key_version: "encryptionkey@0"
I20260927 22:07:24.459731 25162 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.001s
I20260927 22:07:24.466428 25177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:24.469776 25162 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260927 22:07:24.470149 25162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "4a1d6abb3e1543a9bb83a63fc1bec90f"
format_stamp: "Formatted at 2026-09-27 22:07:24 on dist-test-slave-sg4j"
server_key: "4b3571068dbb02edefa995fa59dfe454"
server_key_iv: "dfc4d3dbf80583db0da31e2d52b6ff52"
server_key_version: "encryptionkey@0"
I20260927 22:07:24.470511 25162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:24.512609 25162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:24.519831 25162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:24.520440 25162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:24.549602 25162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44395
I20260927 22:07:24.549695 25228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44395 every 8 connection(s)
I20260927 22:07:24.551867 25162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:24.557235 25229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:24.562016 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 25162
I20260927 22:07:24.562237 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:24.562991 3708 external_mini_cluster.cc:1428] Setting key 611f5b2ca79128c7c583bfd073f5ce7e
I20260927 22:07:24.575977 25229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f: Bootstrap starting.
I20260927 22:07:24.581337 25229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:24.583114 25229 log.cc:824] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:07:24 dist-test-slave-sg4j krb5kdc[25149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 22:07:24.590920 25229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f: No bootstrap required, opened a new log
I20260927 22:07:24.603114 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:24.565951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:34795 (local address 127.0.0.1:44395)
0927 22:07:24.567258 (+ 1307us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:24.567277 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:24.567347 (+ 70us) server_negotiation.cc:407] Connection header received
0927 22:07:24.569714 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:24.569752 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:24.570472 (+ 720us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:24.570998 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:24.573646 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:24.575707 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:24.578627 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:24.579756 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:24.580004 (+ 248us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:24.588714 (+ 8710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:24.588725 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:24.588750 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:24.588829 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:24.594628 (+ 5799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:24.596652 (+ 2024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:24.596664 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:24.596674 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:24.597165 (+ 491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:24.598361 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:24.598366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:24.598369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:24.599284 (+ 915us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:24.599743 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:24.600855 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0927 22:07:24.601497 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":618,"threads_started":1}
I20260927 22:07:24.603609 25229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1d6abb3e1543a9bb83a63fc1bec90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } }
I20260927 22:07:24.604224 25229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:24.604352 25229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1d6abb3e1543a9bb83a63fc1bec90f, State: Initialized, Role: FOLLOWER
I20260927 22:07:24.605406 25229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1d6abb3e1543a9bb83a63fc1bec90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } }
I20260927 22:07:24.605792 25229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:24.606232 25229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:24.606594 25229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:24.609694 25229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1d6abb3e1543a9bb83a63fc1bec90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } }
I20260927 22:07:24.609899 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:24.610399 25229 leader_election.cc:302] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a1d6abb3e1543a9bb83a63fc1bec90f; no voters:
I20260927 22:07:24.611233 25229 leader_election.cc:288] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:24.611513 25234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:24.613862 25229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:24.619220 25234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [term 1 LEADER]: Becoming Leader. State: Replica: 4a1d6abb3e1543a9bb83a63fc1bec90f, State: Running, Role: LEADER
I20260927 22:07:24.620570 25234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1d6abb3e1543a9bb83a63fc1bec90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } }
I20260927 22:07:24.631899 25236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a1d6abb3e1543a9bb83a63fc1bec90f. Latest consensus state: current_term: 1 leader_uuid: "4a1d6abb3e1543a9bb83a63fc1bec90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1d6abb3e1543a9bb83a63fc1bec90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } } }
I20260927 22:07:24.632797 25236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:24.638945 25235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a1d6abb3e1543a9bb83a63fc1bec90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a1d6abb3e1543a9bb83a63fc1bec90f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44395 } } }
I20260927 22:07:24.639488 25235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:24.640379 25255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:24.647111 25255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:24.661409 25255 catalog_manager.cc:1347] Generated new cluster ID: 6f374dce483b4dbd86c8194ceb88ce76
I20260927 22:07:24.661705 25255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:24.661976 25232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:24.632729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:33751 (local address 127.0.0.1:44395)
0927 22:07:24.633078 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:24.633090 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:24.638720 (+ 5630us) server_negotiation.cc:407] Connection header received
0927 22:07:24.639185 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:24.639196 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:24.639335 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:24.639450 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:24.642504 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:24.643908 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:24.646433 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:24.647231 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:24.647431 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:24.652185 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:24.652195 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:24.652201 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:24.652269 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:24.656874 (+ 4605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:24.658801 (+ 1927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:24.658808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:24.658811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:24.659072 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:24.659702 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:24.659708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:24.659710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:24.660218 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:24.660557 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:24.661170 (+ 613us) server_negotiation.cc:299] Negotiation successful
0927 22:07:24.661457 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260927 22:07:24.726241 25255 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:24.728566 25255 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:24.742296 25255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a1d6abb3e1543a9bb83a63fc1bec90f: Generated new TSK 0
I20260927 22:07:24.743511 25255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:24.767931 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25162
I20260927 22:07:24.770614 25224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:24.892278 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 25162
W20260927 22:07:24.915527 25147 connection.cc:535] client connection to 127.0.0.1:44395 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44395): Connection reset by peer (error 104)
2026-09-27T22:07:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2585 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:25 dist-test-slave-sg4j krb5kdc[25262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:25 dist-test-slave-sg4j krb5kdc[25262](info): set up 2 sockets
Sep 27 22:07:25 dist-test-slave-sg4j krb5kdc[25262](info): commencing operation
krb5kdc: starting...
W20260927 22:07:26.066742 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:26 dist-test-slave-sg4j krb5kdc[25262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:26.418229 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:26.859309 25275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:26.859751 25275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:26.859929 25275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:26.868727 25275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:26.868911 25275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:26.869004 25275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:26.869118 25275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:26.869187 25275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:26.881260 25275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43129
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:26.883239 25275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:26.884945 25275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:26.900583 25283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:26.900633 25281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:26.901091 25280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:26.902014 25275 server_base.cc:1034] running on GCE node
I20260927 22:07:26.903786 25275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:26.906826 25275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:26.910302 25275 hybrid_clock.cc:648] HybridClock initialized: now 1790546846910204 us; error 91 us; skew 500 ppm
I20260927 22:07:26.911031 25275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:26 dist-test-slave-sg4j krb5kdc[25262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:26.916411 25275 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:26.918685 25275 webserver.cc:460] Webserver started at http://127.0.0.1:45985/ using document root <none> and password file <none>
I20260927 22:07:26.919622 25275 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:26.919757 25275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:26.920111 25275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:26.924132 25275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "57003cdfee8c415c994025c5e62b9d1e"
format_stamp: "Formatted at 2026-09-27 22:07:26 on dist-test-slave-sg4j"
server_key: "9d88d61665800042623fac2b6330c930"
server_key_iv: "e77d0152627dc543d93a409a6bc51529"
server_key_version: "encryptionkey@0"
I20260927 22:07:26.925097 25275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "57003cdfee8c415c994025c5e62b9d1e"
format_stamp: "Formatted at 2026-09-27 22:07:26 on dist-test-slave-sg4j"
server_key: "9d88d61665800042623fac2b6330c930"
server_key_iv: "e77d0152627dc543d93a409a6bc51529"
server_key_version: "encryptionkey@0"
I20260927 22:07:26.932662 25275 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 22:07:26.938519 25290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:26.940773 25275 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260927 22:07:26.941047 25275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "57003cdfee8c415c994025c5e62b9d1e"
format_stamp: "Formatted at 2026-09-27 22:07:26 on dist-test-slave-sg4j"
server_key: "9d88d61665800042623fac2b6330c930"
server_key_iv: "e77d0152627dc543d93a409a6bc51529"
server_key_version: "encryptionkey@0"
I20260927 22:07:26.941322 25275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:26.969481 25275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:26.975433 25275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:26.975757 25275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:27.000576 25275 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43129
I20260927 22:07:27.000725 25341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43129 every 8 connection(s)
I20260927 22:07:27.003047 25275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:27.005964 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 25275
I20260927 22:07:27.006186 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:27.006704 3708 external_mini_cluster.cc:1428] Setting key b7a2fc3c4faa2a6848158601491ae31a
I20260927 22:07:27.009501 25342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:27.026585 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:27.009593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:56459 (local address 127.0.0.1:43129)
0927 22:07:27.010764 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:27.010781 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:27.010845 (+ 64us) server_negotiation.cc:407] Connection header received
0927 22:07:27.013227 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:27.013264 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:27.013967 (+ 703us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:27.014769 (+ 802us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:27.015657 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:27.018056 (+ 2399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:27.020241 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:27.021454 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:27.021696 (+ 242us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:27.023292 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:27.023304 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:27.023332 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:27.024197 (+ 865us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:27.024334 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:27.024378 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:27.024574 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:27.024926 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 22:07:27.025134 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":494,"threads_started":1}
I20260927 22:07:27.027153 25342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e: Bootstrap starting.
I20260927 22:07:27.031358 25342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:27.032920 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:27.033027 25342 log.cc:824] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:27.037906 25342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e: No bootstrap required, opened a new log
I20260927 22:07:27.047225 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:27.039135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:48739 (local address 127.0.0.1:43129)
0927 22:07:27.039413 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:27.039420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:27.039698 (+ 278us) server_negotiation.cc:407] Connection header received
0927 22:07:27.039993 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:27.039997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:27.040101 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:27.040245 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:27.040884 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:27.041918 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:27.043610 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:27.044201 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:27.044392 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:27.045956 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:27.045963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:27.045966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:27.045994 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:27.046058 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:27.046079 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:27.046217 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:27.046516 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 22:07:27.046592 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 22:07:27.047204 25342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57003cdfee8c415c994025c5e62b9d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260927 22:07:27.047890 25342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:27.048030 25342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57003cdfee8c415c994025c5e62b9d1e, State: Initialized, Role: FOLLOWER
I20260927 22:07:27.049036 25342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57003cdfee8c415c994025c5e62b9d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260927 22:07:27.049402 25342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:27.049585 25342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:27.049801 25342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:27.052263 25342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57003cdfee8c415c994025c5e62b9d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260927 22:07:27.052841 25342 leader_election.cc:302] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57003cdfee8c415c994025c5e62b9d1e; no voters:
I20260927 22:07:27.053753 25342 leader_election.cc:288] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:27.054037 25352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:27.055194 25352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [term 1 LEADER]: Becoming Leader. State: Replica: 57003cdfee8c415c994025c5e62b9d1e, State: Running, Role: LEADER
I20260927 22:07:27.056046 25352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57003cdfee8c415c994025c5e62b9d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } }
I20260927 22:07:27.057313 25342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:27.060133 25354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57003cdfee8c415c994025c5e62b9d1e. Latest consensus state: current_term: 1 leader_uuid: "57003cdfee8c415c994025c5e62b9d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57003cdfee8c415c994025c5e62b9d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } } }
I20260927 22:07:27.060839 25354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:27.061731 25353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57003cdfee8c415c994025c5e62b9d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57003cdfee8c415c994025c5e62b9d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43129 } } }
I20260927 22:07:27.062258 25353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:27.064476 25357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:27.072218 25357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:27.084924 25357 catalog_manager.cc:1347] Generated new cluster ID: 3882761e045c434dbdb510e6db6bf487
I20260927 22:07:27.085107 25357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:27.103314 25357 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:27.105417 25357 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:27.125298 25357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57003cdfee8c415c994025c5e62b9d1e: Generated new TSK 0
I20260927 22:07:27.126583 25357 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:27.143766 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25275
W20260927 22:07:27.145577 25337 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.143:56459
W20260927 22:07:27.146813 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25275: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:27.147060 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 25275
W20260927 22:07:27.169734 25260 connection.cc:535] client connection to 127.0.0.1:43129 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43129): Connection reset by peer (error 104)
2026-09-27T22:07:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:27 dist-test-slave-sg4j krb5kdc[25373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:27 dist-test-slave-sg4j krb5kdc[25373](info): set up 2 sockets
Sep 27 22:07:27 dist-test-slave-sg4j krb5kdc[25373](info): commencing operation
krb5kdc: starting...
W20260927 22:07:28.330499 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.002s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:28 dist-test-slave-sg4j krb5kdc[25373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 22:07:28.683269 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:29.149472 25386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:29.150106 25386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:29.150269 25386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:29.161119 25386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:29.161405 25386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:29.161538 25386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:29.161643 25386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:29.161693 25386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:29.175194 25386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35071
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:29.177106 25386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:29.179147 25386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:29.198249 25392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:29.198105 25394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:29.198083 25391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:29.200624 25386 server_base.cc:1034] running on GCE node
I20260927 22:07:29.201987 25386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:29.204156 25386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:29.205621 25386 hybrid_clock.cc:648] HybridClock initialized: now 1790546849205552 us; error 73 us; skew 500 ppm
I20260927 22:07:29.206211 25386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:29 dist-test-slave-sg4j krb5kdc[25373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:29.214007 25386 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 22:07:29.216507 25386 webserver.cc:460] Webserver started at http://127.0.0.1:37875/ using document root <none> and password file <none>
I20260927 22:07:29.217572 25386 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:29.217717 25386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:29.218047 25386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:29.222683 25386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "862f5a27f2154251a6739d69eaff2aac"
format_stamp: "Formatted at 2026-09-27 22:07:29 on dist-test-slave-sg4j"
server_key: "8c81f7299a47be950f2914f6047edc87"
server_key_iv: "49e2faf8d0bd486c166e6de4b8b33703"
server_key_version: "encryptionkey@0"
I20260927 22:07:29.223881 25386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "862f5a27f2154251a6739d69eaff2aac"
format_stamp: "Formatted at 2026-09-27 22:07:29 on dist-test-slave-sg4j"
server_key: "8c81f7299a47be950f2914f6047edc87"
server_key_iv: "49e2faf8d0bd486c166e6de4b8b33703"
server_key_version: "encryptionkey@0"
I20260927 22:07:29.231736 25386 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260927 22:07:29.237643 25401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:29.239934 25386 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260927 22:07:29.240276 25386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "862f5a27f2154251a6739d69eaff2aac"
format_stamp: "Formatted at 2026-09-27 22:07:29 on dist-test-slave-sg4j"
server_key: "8c81f7299a47be950f2914f6047edc87"
server_key_iv: "49e2faf8d0bd486c166e6de4b8b33703"
server_key_version: "encryptionkey@0"
I20260927 22:07:29.240577 25386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:29.248492 25386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:29.254410 25386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:29.254799 25386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:29.281095 25386 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35071
I20260927 22:07:29.281179 25452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35071 every 8 connection(s)
I20260927 22:07:29.283210 25386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:29.289431 25453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:29.292204 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 25386
I20260927 22:07:29.292651 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:29.293754 3708 external_mini_cluster.cc:1428] Setting key a6abdd03b06d94bf25033edc2e54f6ad
I20260927 22:07:29.307140 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:29.296993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:40275 (local address 127.0.0.1:35071)
0927 22:07:29.299087 (+ 2094us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:29.299107 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:29.299201 (+ 94us) server_negotiation.cc:407] Connection header received
0927 22:07:29.301458 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:29.301515 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:29.302196 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:29.302833 (+ 637us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:29.304477 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:29.304485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:29.304504 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:29.304521 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:29.304599 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:29.304624 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:29.304939 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:29.305129 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 22:07:29.305595 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1500,"thread_start_us":1035,"threads_started":1}
I20260927 22:07:29.310791 25453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac: Bootstrap starting.
I20260927 22:07:29.314134 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:29.318243 25453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:29.319900 25453 log.cc:824] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:29.324402 25456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:29.320393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:48715 (local address 127.0.0.1:35071)
0927 22:07:29.320717 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:29.320728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:29.321466 (+ 738us) server_negotiation.cc:407] Connection header received
0927 22:07:29.321842 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:29.321847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:29.321924 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:29.322060 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:29.323614 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:29.323619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:29.323622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:29.323645 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:29.323701 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:29.323719 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:29.323862 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:29.324059 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 22:07:29.324129 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:07:29.325452 25453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac: No bootstrap required, opened a new log
I20260927 22:07:29.337116 25453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f5a27f2154251a6739d69eaff2aac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260927 22:07:29.337879 25453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:29.337983 25453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862f5a27f2154251a6739d69eaff2aac, State: Initialized, Role: FOLLOWER
I20260927 22:07:29.338848 25453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f5a27f2154251a6739d69eaff2aac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260927 22:07:29.339149 25453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:29.339269 25453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:29.339452 25453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:29.343410 25453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f5a27f2154251a6739d69eaff2aac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260927 22:07:29.344178 25453 leader_election.cc:302] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 862f5a27f2154251a6739d69eaff2aac; no voters:
I20260927 22:07:29.344915 25453 leader_election.cc:288] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:29.345150 25463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:29.346402 25463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [term 1 LEADER]: Becoming Leader. State: Replica: 862f5a27f2154251a6739d69eaff2aac, State: Running, Role: LEADER
I20260927 22:07:29.347177 25463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f5a27f2154251a6739d69eaff2aac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } }
I20260927 22:07:29.347990 25453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:29.352981 25464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "862f5a27f2154251a6739d69eaff2aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f5a27f2154251a6739d69eaff2aac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } } }
I20260927 22:07:29.353533 25464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:29.354127 25465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 862f5a27f2154251a6739d69eaff2aac. Latest consensus state: current_term: 1 leader_uuid: "862f5a27f2154251a6739d69eaff2aac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "862f5a27f2154251a6739d69eaff2aac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35071 } } }
I20260927 22:07:29.354830 25465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:29.357987 25472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:29.368095 25472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:29.385807 25472 catalog_manager.cc:1347] Generated new cluster ID: 306e52d86f154afe87b9fdc4d067f3cd
I20260927 22:07:29.385962 25472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:29.406026 25472 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:29.407531 25472 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:29.425093 25472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 862f5a27f2154251a6739d69eaff2aac: Generated new TSK 0
I20260927 22:07:29.427109 25472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:29.446775 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25386
W20260927 22:07:29.448948 25448 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.143:40275
W20260927 22:07:29.450358 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25386: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:29.450680 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 25386
2026-09-27T22:07:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:29 dist-test-slave-sg4j krb5kdc[25484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:29 dist-test-slave-sg4j krb5kdc[25484](info): set up 2 sockets
Sep 27 22:07:29 dist-test-slave-sg4j krb5kdc[25484](info): commencing operation
krb5kdc: starting...
W20260927 22:07:30.670594 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:30 dist-test-slave-sg4j krb5kdc[25484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:07:30.996843 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:36373
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:34569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:36373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:31.408257 25497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:31.408681 25497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:31.408869 25497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:31.417100 25497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:31.417256 25497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:31.417315 25497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:31.417403 25497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:31.417447 25497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:31.429409 25497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:34569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:36373
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:36373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:31.431360 25497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:31.433171 25497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:31.448246 25505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:31.448294 25503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:31.448161 25502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:31.449685 25497 server_base.cc:1034] running on GCE node
I20260927 22:07:31.450811 25497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:31.452618 25497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:31.453886 25497 hybrid_clock.cc:648] HybridClock initialized: now 1790546851453824 us; error 49 us; skew 500 ppm
I20260927 22:07:31.454387 25497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:31 dist-test-slave-sg4j krb5kdc[25484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:31.458429 25497 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:07:31.460716 25497 webserver.cc:460] Webserver started at http://127.3.159.62:46037/ using document root <none> and password file <none>
I20260927 22:07:31.461691 25497 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:31.461862 25497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:31.462232 25497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:31.465914 25497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "6ca48d2b6c19411a9610c58caa32284a"
format_stamp: "Formatted at 2026-09-27 22:07:31 on dist-test-slave-sg4j"
server_key: "0c75d562d9af9d94ca28c7cd46f5296d"
server_key_iv: "df1e70a3d5ada8729bc2999d9ba2f577"
server_key_version: "encryptionkey@0"
I20260927 22:07:31.466804 25497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "6ca48d2b6c19411a9610c58caa32284a"
format_stamp: "Formatted at 2026-09-27 22:07:31 on dist-test-slave-sg4j"
server_key: "0c75d562d9af9d94ca28c7cd46f5296d"
server_key_iv: "df1e70a3d5ada8729bc2999d9ba2f577"
server_key_version: "encryptionkey@0"
I20260927 22:07:31.473259 25497 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260927 22:07:31.477882 25512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:31.480223 25497 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 22:07:31.480551 25497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "6ca48d2b6c19411a9610c58caa32284a"
format_stamp: "Formatted at 2026-09-27 22:07:31 on dist-test-slave-sg4j"
server_key: "0c75d562d9af9d94ca28c7cd46f5296d"
server_key_iv: "df1e70a3d5ada8729bc2999d9ba2f577"
server_key_version: "encryptionkey@0"
I20260927 22:07:31.480898 25497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:31.504798 25497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:31.511006 25497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:31.511404 25497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:31.535734 25497 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:36373
I20260927 22:07:31.535845 25563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:36373 every 8 connection(s)
I20260927 22:07:31.537354 25497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:31.540987 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 25497
I20260927 22:07:31.541221 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:31.541798 3708 external_mini_cluster.cc:1428] Setting key 265fff48f385b7bee002ede76cdf0347
I20260927 22:07:31.542672 25564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 22:07:31 dist-test-slave-sg4j krb5kdc[25484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:07:31.560577 25564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a: Bootstrap starting.
I20260927 22:07:31.565282 25564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:31.566831 25564 log.cc:824] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:31.571450 25564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a: No bootstrap required, opened a new log
I20260927 22:07:31.573108 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:31.544160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58471 (local address 127.3.159.62:36373)
0927 22:07:31.545086 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:31.545100 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:31.545150 (+ 50us) server_negotiation.cc:407] Connection header received
0927 22:07:31.546902 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:31.546928 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:31.547421 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:31.547806 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:31.549990 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:31.551617 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:31.553632 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:31.554686 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:31.555025 (+ 339us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:31.561246 (+ 6221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:31.561259 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:31.561280 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:31.561339 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:31.566443 (+ 5104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:31.567854 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:31.567864 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:31.567872 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:31.568112 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:31.569069 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:31.569077 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:31.569081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:31.570389 (+ 1308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:31.570834 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:31.571440 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 22:07:31.571863 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":433,"threads_started":1}
I20260927 22:07:31.579373 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:31.582444 25564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca48d2b6c19411a9610c58caa32284a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36373 } }
I20260927 22:07:31.583184 25564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:31.583348 25564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca48d2b6c19411a9610c58caa32284a, State: Initialized, Role: FOLLOWER
I20260927 22:07:31.584249 25564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca48d2b6c19411a9610c58caa32284a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36373 } }
I20260927 22:07:31.584571 25564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:31.584774 25564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:31.585000 25564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:31.588424 25564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca48d2b6c19411a9610c58caa32284a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36373 } }
I20260927 22:07:31.589243 25564 leader_election.cc:302] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ca48d2b6c19411a9610c58caa32284a; no voters:
I20260927 22:07:31.590265 25564 leader_election.cc:288] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:31.590421 25574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:31.594101 25564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:31.597885 25574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [term 1 LEADER]: Becoming Leader. State: Replica: 6ca48d2b6c19411a9610c58caa32284a, State: Running, Role: LEADER
I20260927 22:07:31.599107 25574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca48d2b6c19411a9610c58caa32284a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36373 } }
I20260927 22:07:31.603675 25576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ca48d2b6c19411a9610c58caa32284a. Latest consensus state: current_term: 1 leader_uuid: "6ca48d2b6c19411a9610c58caa32284a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca48d2b6c19411a9610c58caa32284a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36373 } } }
I20260927 22:07:31.604278 25576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:31.606948 25575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ca48d2b6c19411a9610c58caa32284a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca48d2b6c19411a9610c58caa32284a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 36373 } } }
I20260927 22:07:31.607077 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:31.586028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56855 (local address 127.3.159.62:36373)
0927 22:07:31.586217 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:31.586222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:31.586907 (+ 685us) server_negotiation.cc:407] Connection header received
0927 22:07:31.587362 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:31.587366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:31.587434 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:31.587577 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:31.589338 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:31.590301 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:31.592160 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:31.592774 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:31.592936 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:31.596696 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:31.596702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:31.596706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:31.596752 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:31.600605 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:31.601809 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:31.601827 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:31.601831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:31.602195 (+ 364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:31.602858 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:31.602864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:31.602867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:31.603271 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:31.603609 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:31.605056 (+ 1447us) server_negotiation.cc:299] Negotiation successful
0927 22:07:31.605323 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 22:07:31.607568 25575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:31.610388 25581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:31.615585 25581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:31.627163 25581 catalog_manager.cc:1347] Generated new cluster ID: 3589e390c6dd4392918ed6481bc80962
I20260927 22:07:31.627326 25581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:31.638477 25581 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:31.639925 25581 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:31.650031 25581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ca48d2b6c19411a9610c58caa32284a: Generated new TSK 0
I20260927 22:07:31.651238 25581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:31.661646 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25497
I20260927 22:07:31.663347 25559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:31.776968 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 25497
W20260927 22:07:31.797601 25482 connection.cc:535] client connection to 127.3.159.62:36373 recv error: Network error: failed to read from TLS socket (remote: 127.3.159.62:36373): Connection reset by peer (error 104)
2026-09-27T22:07:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:31 dist-test-slave-sg4j krb5kdc[25597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:31 dist-test-slave-sg4j krb5kdc[25597](info): set up 2 sockets
Sep 27 22:07:31 dist-test-slave-sg4j krb5kdc[25597](info): commencing operation
krb5kdc: starting...
W20260927 22:07:32.998732 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:33 dist-test-slave-sg4j krb5kdc[25597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:07:33.351773 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37227
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:39761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:37227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:33.800760 25610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:33.801208 25610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:33.801436 25610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:33.811273 25610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:33.811473 25610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:33.811570 25610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:33.811704 25610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:33.811780 25610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:33.824083 25610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:39761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:37227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:33.826161 25610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:33.828085 25610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:33.845498 25615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:33.845469 25616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:33.845538 25618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:33.846835 25610 server_base.cc:1034] running on GCE node
I20260927 22:07:33.847869 25610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:33.849836 25610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:33.853484 25610 hybrid_clock.cc:648] HybridClock initialized: now 1790546853853200 us; error 145 us; skew 500 ppm
I20260927 22:07:33.854077 25610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:33 dist-test-slave-sg4j krb5kdc[25597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546853, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:33.858491 25610 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:07:33.860762 25610 webserver.cc:460] Webserver started at http://127.3.159.62:44007/ using document root <none> and password file <none>
I20260927 22:07:33.861780 25610 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:33.861965 25610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:33.862346 25610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:33.866255 25610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "cdf6069e52924b2abc223842538b8d1a"
format_stamp: "Formatted at 2026-09-27 22:07:33 on dist-test-slave-sg4j"
server_key: "2f83ece1711d45ab4b494e1ca0749a45"
server_key_iv: "eb12fa707dd4c336600cc0caefd2fc42"
server_key_version: "encryptionkey@0"
I20260927 22:07:33.867419 25610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "cdf6069e52924b2abc223842538b8d1a"
format_stamp: "Formatted at 2026-09-27 22:07:33 on dist-test-slave-sg4j"
server_key: "2f83ece1711d45ab4b494e1ca0749a45"
server_key_iv: "eb12fa707dd4c336600cc0caefd2fc42"
server_key_version: "encryptionkey@0"
I20260927 22:07:33.875936 25610 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260927 22:07:33.881958 25625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:33.884511 25610 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:07:33.884878 25610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "cdf6069e52924b2abc223842538b8d1a"
format_stamp: "Formatted at 2026-09-27 22:07:33 on dist-test-slave-sg4j"
server_key: "2f83ece1711d45ab4b494e1ca0749a45"
server_key_iv: "eb12fa707dd4c336600cc0caefd2fc42"
server_key_version: "encryptionkey@0"
I20260927 22:07:33.885190 25610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:33.921973 25610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:33.927043 25610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:33.927380 25610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:33.954567 25610 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:37227
I20260927 22:07:33.954687 25676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:37227 every 8 connection(s)
I20260927 22:07:33.956233 25610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:33.961586 25677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:33.963137 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 25610
I20260927 22:07:33.963369 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:33.963835 3708 external_mini_cluster.cc:1428] Setting key 05a9c6cb5b376f81616364368a5eb06f
I20260927 22:07:33.980792 25677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a: Bootstrap starting.
I20260927 22:07:33.983023 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:33.966524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41291 (local address 127.3.159.62:37227)
0927 22:07:33.967648 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:33.967661 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:33.967722 (+ 61us) server_negotiation.cc:407] Connection header received
0927 22:07:33.970527 (+ 2805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:33.970564 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:33.971175 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:33.971643 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:33.972645 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:33.974279 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:33.976625 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:33.977758 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:33.977963 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:33.979710 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:33.979721 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:33.979744 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:33.980689 (+ 945us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:33.980821 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:33.980857 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:33.981112 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:33.981563 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 22:07:33.981784 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":536,"threads_started":1}
I20260927 22:07:33.985423 25677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:33.987018 25677 log.cc:824] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:33.988087 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:33.990914 25677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a: No bootstrap required, opened a new log
I20260927 22:07:34.000962 25677 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6069e52924b2abc223842538b8d1a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37227 } }
I20260927 22:07:34.001812 25677 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:34.001965 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:33.994049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37021 (local address 127.3.159.62:37227)
0927 22:07:33.994389 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:33.994401 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:33.995385 (+ 984us) server_negotiation.cc:407] Connection header received
0927 22:07:33.995645 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:33.995650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:33.995736 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:33.995873 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:33.996454 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:33.997292 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:33.998758 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:33.999250 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:33.999413 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:34.000664 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:34.000669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:34.000672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:34.000694 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:34.000760 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:34.000778 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:34.000907 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:34.001103 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 22:07:34.001165 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 22:07:34.001968 25677 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdf6069e52924b2abc223842538b8d1a, State: Initialized, Role: FOLLOWER
I20260927 22:07:34.003073 25677 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6069e52924b2abc223842538b8d1a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37227 } }
I20260927 22:07:34.003403 25677 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:34.003574 25677 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:34.003829 25677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:34.006639 25677 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6069e52924b2abc223842538b8d1a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37227 } }
I20260927 22:07:34.007292 25677 leader_election.cc:302] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdf6069e52924b2abc223842538b8d1a; no voters:
I20260927 22:07:34.008101 25677 leader_election.cc:288] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:34.008435 25687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:34.009879 25687 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [term 1 LEADER]: Becoming Leader. State: Replica: cdf6069e52924b2abc223842538b8d1a, State: Running, Role: LEADER
I20260927 22:07:34.010728 25687 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6069e52924b2abc223842538b8d1a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37227 } }
I20260927 22:07:34.011759 25677 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:34.014278 25689 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdf6069e52924b2abc223842538b8d1a. Latest consensus state: current_term: 1 leader_uuid: "cdf6069e52924b2abc223842538b8d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6069e52924b2abc223842538b8d1a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37227 } } }
I20260927 22:07:34.014256 25688 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdf6069e52924b2abc223842538b8d1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdf6069e52924b2abc223842538b8d1a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37227 } } }
I20260927 22:07:34.014860 25689 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:34.014935 25688 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:34.015934 25692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:34.027093 25692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:34.039467 25692 catalog_manager.cc:1347] Generated new cluster ID: 8c1129366ecb4a9eb61f4c954f12eb88
I20260927 22:07:34.039644 25692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:34.066553 25692 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:34.068033 25692 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:34.076879 25692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdf6069e52924b2abc223842538b8d1a: Generated new TSK 0
I20260927 22:07:34.078058 25692 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:34.090116 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25610
W20260927 22:07:34.091506 25672 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41291
W20260927 22:07:34.092559 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25610: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:34.092749 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 25610
W20260927 22:07:34.111905 25595 connection.cc:535] client connection to 127.3.159.62:37227 recv error: Network error: failed to read from TLS socket (remote: 127.3.159.62:37227): Connection reset by peer (error 104)
2026-09-27T22:07:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:34 dist-test-slave-sg4j krb5kdc[25708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:34 dist-test-slave-sg4j krb5kdc[25708](info): set up 2 sockets
Sep 27 22:07:34 dist-test-slave-sg4j krb5kdc[25708](info): commencing operation
krb5kdc: starting...
W20260927 22:07:35.326453 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.005s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:35 dist-test-slave-sg4j krb5kdc[25708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:07:35.666179 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:42865
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:42865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:36.088783 25721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:36.089200 25721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:36.089406 25721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:36.098359 25721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:36.098590 25721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:36.098722 25721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:36.098845 25721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:36.098928 25721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:36.109920 25721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:42865
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:42865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:36.111397 25721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:36.112810 25721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:36.127863 25726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:36.127923 25729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:36.127978 25727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:36.129537 25721 server_base.cc:1034] running on GCE node
I20260927 22:07:36.130517 25721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:36.132243 25721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:36.133723 25721 hybrid_clock.cc:648] HybridClock initialized: now 1790546856133652 us; error 103 us; skew 500 ppm
I20260927 22:07:36.134200 25721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:36 dist-test-slave-sg4j krb5kdc[25708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:36.144153 25721 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:07:36.146313 25721 webserver.cc:460] Webserver started at http://127.3.159.62:37809/ using document root <none> and password file <none>
I20260927 22:07:36.147135 25721 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:36.147289 25721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:36.147650 25721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:36.151150 25721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "3dcb3e7f83e246b9a9f6834301d3d8c6"
format_stamp: "Formatted at 2026-09-27 22:07:36 on dist-test-slave-sg4j"
server_key: "5f0a98cfca2ff1df36f7bd0133069131"
server_key_iv: "520c2c3fb2a69cf805f04954b5d03e4f"
server_key_version: "encryptionkey@0"
I20260927 22:07:36.152309 25721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "3dcb3e7f83e246b9a9f6834301d3d8c6"
format_stamp: "Formatted at 2026-09-27 22:07:36 on dist-test-slave-sg4j"
server_key: "5f0a98cfca2ff1df36f7bd0133069131"
server_key_iv: "520c2c3fb2a69cf805f04954b5d03e4f"
server_key_version: "encryptionkey@0"
I20260927 22:07:36.159029 25721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 22:07:36.165056 25736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:36.167160 25721 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 22:07:36.167449 25721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "3dcb3e7f83e246b9a9f6834301d3d8c6"
format_stamp: "Formatted at 2026-09-27 22:07:36 on dist-test-slave-sg4j"
server_key: "5f0a98cfca2ff1df36f7bd0133069131"
server_key_iv: "520c2c3fb2a69cf805f04954b5d03e4f"
server_key_version: "encryptionkey@0"
I20260927 22:07:36.167711 25721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:36.175424 25721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:36.180838 25721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:36.181247 25721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:36.205731 25721 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:42865
I20260927 22:07:36.205940 25787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:42865 every 8 connection(s)
I20260927 22:07:36.207361 25721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:36.211694 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 25721
I20260927 22:07:36.211856 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:36.212391 3708 external_mini_cluster.cc:1428] Setting key 7520b2e5e005dbf51cdd972b192cbb1b
I20260927 22:07:36.212730 25788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:36.223153 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:36.214502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53333 (local address 127.3.159.62:42865)
0927 22:07:36.215442 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:36.215456 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:36.215505 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:07:36.218452 (+ 2947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:36.218482 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:36.218938 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:36.219362 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:36.220860 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:36.220871 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:36.220891 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:36.220913 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:36.220996 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:36.221027 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:36.221253 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:36.221443 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 22:07:36.221754 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":413,"threads_started":1}
I20260927 22:07:36.229199 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:36.230022 25788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6: Bootstrap starting.
I20260927 22:07:36.234345 25788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:36.236227 25788 log.cc:824] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:36.238628 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:36.235548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36229 (local address 127.3.159.62:42865)
0927 22:07:36.235814 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:36.235827 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:36.236155 (+ 328us) server_negotiation.cc:407] Connection header received
0927 22:07:36.236456 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:36.236464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:36.236546 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:36.236668 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:36.237862 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:36.237872 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:36.237878 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:36.237900 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:36.237958 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:36.237976 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:36.238122 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:36.238214 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 22:07:36.238304 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 22:07:36.240684 25788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6: No bootstrap required, opened a new log
I20260927 22:07:36.249747 25788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcb3e7f83e246b9a9f6834301d3d8c6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42865 } }
I20260927 22:07:36.250298 25788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:36.250429 25788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dcb3e7f83e246b9a9f6834301d3d8c6, State: Initialized, Role: FOLLOWER
I20260927 22:07:36.251176 25788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcb3e7f83e246b9a9f6834301d3d8c6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42865 } }
I20260927 22:07:36.251468 25788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:36.251642 25788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:36.251835 25788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:36.254784 25788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcb3e7f83e246b9a9f6834301d3d8c6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42865 } }
I20260927 22:07:36.255465 25788 leader_election.cc:302] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dcb3e7f83e246b9a9f6834301d3d8c6; no voters:
I20260927 22:07:36.256145 25788 leader_election.cc:288] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:36.256417 25798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:36.257858 25798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [term 1 LEADER]: Becoming Leader. State: Replica: 3dcb3e7f83e246b9a9f6834301d3d8c6, State: Running, Role: LEADER
I20260927 22:07:36.258538 25798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcb3e7f83e246b9a9f6834301d3d8c6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42865 } }
I20260927 22:07:36.258900 25788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:36.262212 25800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dcb3e7f83e246b9a9f6834301d3d8c6. Latest consensus state: current_term: 1 leader_uuid: "3dcb3e7f83e246b9a9f6834301d3d8c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcb3e7f83e246b9a9f6834301d3d8c6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42865 } } }
I20260927 22:07:36.266999 25800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:36.262306 25799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dcb3e7f83e246b9a9f6834301d3d8c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dcb3e7f83e246b9a9f6834301d3d8c6" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 42865 } } }
I20260927 22:07:36.267597 25799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:36.270358 25807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:36.279356 25807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:36.290244 25807 catalog_manager.cc:1347] Generated new cluster ID: dd6dc92a5bda4d5eac594ea702eb421a
I20260927 22:07:36.290406 25807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:36.321411 25807 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:36.322212 25807 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:36.334394 25807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dcb3e7f83e246b9a9f6834301d3d8c6: Generated new TSK 0
I20260927 22:07:36.335907 25807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:36.341429 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25721
W20260927 22:07:36.343147 25783 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53333
W20260927 22:07:36.343983 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:36.344143 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 25721
2026-09-27T22:07:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:36 dist-test-slave-sg4j krb5kdc[25819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:36 dist-test-slave-sg4j krb5kdc[25819](info): set up 2 sockets
Sep 27 22:07:36 dist-test-slave-sg4j krb5kdc[25819](info): commencing operation
krb5kdc: starting...
W20260927 22:07:37.498294 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:37 dist-test-slave-sg4j krb5kdc[25819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:07:37.834400 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:45097
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:45097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:38.220665 25832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:38.221004 25832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:38.221148 25832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:38.228901 25832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:38.229048 25832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:38.229099 25832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:38.229238 25832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:38.229310 25832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:38.239589 25832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:45097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:45097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:38.241343 25832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:38.242920 25832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:38.258026 25838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:38.257980 25840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:38.258106 25837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:38.259151 25832 server_base.cc:1034] running on GCE node
I20260927 22:07:38.260052 25832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:38.262066 25832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:38.263412 25832 hybrid_clock.cc:648] HybridClock initialized: now 1790546858263340 us; error 93 us; skew 500 ppm
I20260927 22:07:38.263911 25832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:38 dist-test-slave-sg4j krb5kdc[25819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546858, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:38.268019 25832 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:07:38.270367 25832 webserver.cc:460] Webserver started at http://127.3.159.62:33585/ using document root <none> and password file <none>
I20260927 22:07:38.271198 25832 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:38.271399 25832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:38.271697 25832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:38.275269 25832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "cda53abf7535411e8e5a14f844c488df"
format_stamp: "Formatted at 2026-09-27 22:07:38 on dist-test-slave-sg4j"
server_key: "34bf0fe63894b89e3c99a585ffc8157e"
server_key_iv: "23e86198d912819b534e60111c3088d9"
server_key_version: "encryptionkey@0"
I20260927 22:07:38.276137 25832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "cda53abf7535411e8e5a14f844c488df"
format_stamp: "Formatted at 2026-09-27 22:07:38 on dist-test-slave-sg4j"
server_key: "34bf0fe63894b89e3c99a585ffc8157e"
server_key_iv: "23e86198d912819b534e60111c3088d9"
server_key_version: "encryptionkey@0"
I20260927 22:07:38.282801 25832 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 22:07:38.287927 25847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:38.290061 25832 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:07:38.290340 25832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "cda53abf7535411e8e5a14f844c488df"
format_stamp: "Formatted at 2026-09-27 22:07:38 on dist-test-slave-sg4j"
server_key: "34bf0fe63894b89e3c99a585ffc8157e"
server_key_iv: "23e86198d912819b534e60111c3088d9"
server_key_version: "encryptionkey@0"
I20260927 22:07:38.290727 25832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:38.311748 25832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:38.316638 25832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:38.317003 25832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:38.341637 25832 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:45097
I20260927 22:07:38.341748 25898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:45097 every 8 connection(s)
I20260927 22:07:38.343935 25832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:38.349370 25899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:38.350337 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 25832
I20260927 22:07:38.350824 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:38.351328 3708 external_mini_cluster.cc:1428] Setting key 1e9525cc12be92b416b38fafd5e23f54
I20260927 22:07:38.365203 25899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df: Bootstrap starting.
I20260927 22:07:38.368759 25899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df: Neither blocks nor log segments found. Creating new log.
Sep 27 22:07:38 dist-test-slave-sg4j krb5kdc[25819](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:07:38.370039 25899 log.cc:824] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:38.374428 25899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df: No bootstrap required, opened a new log
I20260927 22:07:38.384209 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:38.354057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42219 (local address 127.3.159.62:45097)
0927 22:07:38.355249 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:38.355268 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:38.355339 (+ 71us) server_negotiation.cc:407] Connection header received
0927 22:07:38.357625 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:38.357668 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:38.358437 (+ 769us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:38.359031 (+ 594us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:38.360800 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:38.362554 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:38.364090 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:38.364958 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:38.365215 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:38.370435 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:38.370447 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:38.370471 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:38.370535 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:38.377239 (+ 6704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:38.378430 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:38.378447 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:38.378466 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:38.378787 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:38.379412 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:38.379417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:38.379420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:38.380385 (+ 965us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:38.380807 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:38.381552 (+ 745us) server_negotiation.cc:299] Negotiation successful
0927 22:07:38.382286 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":565,"threads_started":1}
I20260927 22:07:38.386775 25899 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda53abf7535411e8e5a14f844c488df" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45097 } }
I20260927 22:07:38.387300 25899 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:38.387480 25899 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda53abf7535411e8e5a14f844c488df, State: Initialized, Role: FOLLOWER
I20260927 22:07:38.388377 25899 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda53abf7535411e8e5a14f844c488df" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45097 } }
I20260927 22:07:38.388711 25899 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:38.388877 25899 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:38.389076 25899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:38.390905 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:38.392340 25899 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda53abf7535411e8e5a14f844c488df" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45097 } }
I20260927 22:07:38.393102 25899 leader_election.cc:302] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda53abf7535411e8e5a14f844c488df; no voters:
I20260927 22:07:38.394133 25899 leader_election.cc:288] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:38.394449 25904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:38.396270 25904 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [term 1 LEADER]: Becoming Leader. State: Replica: cda53abf7535411e8e5a14f844c488df, State: Running, Role: LEADER
I20260927 22:07:38.397472 25899 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:38.397406 25904 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda53abf7535411e8e5a14f844c488df" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45097 } }
I20260927 22:07:38.406582 25908 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda53abf7535411e8e5a14f844c488df. Latest consensus state: current_term: 1 leader_uuid: "cda53abf7535411e8e5a14f844c488df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda53abf7535411e8e5a14f844c488df" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45097 } } }
I20260927 22:07:38.407482 25908 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:38.408923 25906 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda53abf7535411e8e5a14f844c488df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda53abf7535411e8e5a14f844c488df" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 45097 } } }
I20260927 22:07:38.409325 25906 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df [sys.catalog]: This master's current role is: LEADER
W20260927 22:07:38.425570 25924 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:07:38.425900 25924 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:07:38.426087 25923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:38.429719 25923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:38.437109 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:38.408703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42749 (local address 127.3.159.62:45097)
0927 22:07:38.408965 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:38.408973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:38.411061 (+ 2088us) server_negotiation.cc:407] Connection header received
0927 22:07:38.411448 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:38.411455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:38.411545 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:38.411645 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:38.415127 (+ 3482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:38.417215 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:38.420390 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:38.422483 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:38.422694 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:38.427312 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:38.427325 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:38.427330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:38.427402 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:38.432008 (+ 4606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:38.433161 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:38.433166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:38.433170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:38.433346 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:38.434073 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:38.434078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:38.434082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:38.434584 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:38.436185 (+ 1601us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:38.436418 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 22:07:38.436718 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 22:07:38.444092 25923 catalog_manager.cc:1347] Generated new cluster ID: 58aefef354c94fa284bf927c79141c11
I20260927 22:07:38.444324 25923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:38.472262 25923 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:38.474684 25923 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:38.493346 25923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda53abf7535411e8e5a14f844c488df: Generated new TSK 0
I20260927 22:07:38.494937 25923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:38.504997 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25832
I20260927 22:07:38.506312 25894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:38.618083 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 25832
2026-09-27T22:07:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2274 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:38 dist-test-slave-sg4j krb5kdc[25931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:38 dist-test-slave-sg4j krb5kdc[25931](info): set up 2 sockets
Sep 27 22:07:38 dist-test-slave-sg4j krb5kdc[25931](info): commencing operation
krb5kdc: starting...
W20260927 22:07:39.802472 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:40 dist-test-slave-sg4j krb5kdc[25931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:07:40.166926 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:35797
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:35797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:40.639084 25944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:40.639580 25944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:40.639781 25944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:40.648272 25944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:40.648537 25944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:40.648669 25944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:40.648779 25944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:40.648876 25944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:40.661019 25944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:35797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:35797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:40.662948 25944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:40.664716 25944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:40.680719 25949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:40.681049 25952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:40.680709 25950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:40.687878 25944 server_base.cc:1034] running on GCE node
I20260927 22:07:40.689050 25944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:40.691131 25944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:40.692713 25944 hybrid_clock.cc:648] HybridClock initialized: now 1790546860692615 us; error 93 us; skew 500 ppm
I20260927 22:07:40.693248 25944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:40 dist-test-slave-sg4j krb5kdc[25931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546860, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:40.697489 25944 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:07:40.699496 25944 webserver.cc:460] Webserver started at http://127.3.159.62:36029/ using document root <none> and password file <none>
I20260927 22:07:40.700371 25944 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:40.700529 25944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:40.700873 25944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:40.704375 25944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "fa748015ea2045fcbd9bc70470e0324a"
format_stamp: "Formatted at 2026-09-27 22:07:40 on dist-test-slave-sg4j"
server_key: "c15bb0d1bb50b2ba28f4d36babaf3eb9"
server_key_iv: "8d060fbae786f2b47e1f4a4ceaa95cf7"
server_key_version: "encryptionkey@0"
I20260927 22:07:40.705276 25944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "fa748015ea2045fcbd9bc70470e0324a"
format_stamp: "Formatted at 2026-09-27 22:07:40 on dist-test-slave-sg4j"
server_key: "c15bb0d1bb50b2ba28f4d36babaf3eb9"
server_key_iv: "8d060fbae786f2b47e1f4a4ceaa95cf7"
server_key_version: "encryptionkey@0"
I20260927 22:07:40.712651 25944 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260927 22:07:40.717834 25959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:40.720089 25944 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260927 22:07:40.720402 25944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "fa748015ea2045fcbd9bc70470e0324a"
format_stamp: "Formatted at 2026-09-27 22:07:40 on dist-test-slave-sg4j"
server_key: "c15bb0d1bb50b2ba28f4d36babaf3eb9"
server_key_iv: "8d060fbae786f2b47e1f4a4ceaa95cf7"
server_key_version: "encryptionkey@0"
I20260927 22:07:40.720669 25944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:40.758260 25944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:40.765529 25944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:40.765951 25944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:40.794101 25944 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:35797
I20260927 22:07:40.794157 26010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:35797 every 8 connection(s)
I20260927 22:07:40.795912 25944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:40.799286 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 25944
I20260927 22:07:40.799531 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:40.800048 3708 external_mini_cluster.cc:1428] Setting key eb719afb917a989002def94181851493
I20260927 22:07:40.802095 26011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:40.818495 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:40.802652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43151 (local address 127.3.159.62:35797)
0927 22:07:40.803689 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:40.803703 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:40.803754 (+ 51us) server_negotiation.cc:407] Connection header received
0927 22:07:40.805828 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:40.805859 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:40.806419 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:40.806841 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:40.807715 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:40.809513 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:40.811752 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:40.812708 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:40.812925 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:40.814776 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:40.814787 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:40.814808 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:40.815569 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:40.815683 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:40.815718 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:40.815862 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:40.816258 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 22:07:40.816790 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":471,"threads_started":1}
I20260927 22:07:40.819624 26011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a: Bootstrap starting.
I20260927 22:07:40.824465 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:40.824539 26011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:40.826268 26011 log.cc:824] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:40.831110 26011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a: No bootstrap required, opened a new log
I20260927 22:07:40.840677 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:40.830036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54945 (local address 127.3.159.62:35797)
0927 22:07:40.830345 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:40.830356 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:40.830955 (+ 599us) server_negotiation.cc:407] Connection header received
0927 22:07:40.831454 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:40.831461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:40.831574 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:40.831722 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:40.832496 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:40.833802 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:40.837068 (+ 3266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:40.837955 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:40.838180 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:40.839572 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:40.839585 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:40.839591 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:40.839625 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:40.839708 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:40.839735 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:40.839881 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:40.840098 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 22:07:40.840248 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 22:07:40.844214 26011 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa748015ea2045fcbd9bc70470e0324a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35797 } }
I20260927 22:07:40.844777 26011 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:40.844930 26011 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa748015ea2045fcbd9bc70470e0324a, State: Initialized, Role: FOLLOWER
I20260927 22:07:40.845873 26011 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa748015ea2045fcbd9bc70470e0324a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35797 } }
I20260927 22:07:40.846199 26011 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:40.846387 26011 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:40.846632 26011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:40.850028 26011 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa748015ea2045fcbd9bc70470e0324a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35797 } }
I20260927 22:07:40.850826 26011 leader_election.cc:302] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa748015ea2045fcbd9bc70470e0324a; no voters:
I20260927 22:07:40.851656 26011 leader_election.cc:288] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:40.851986 26021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:40.853216 26021 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [term 1 LEADER]: Becoming Leader. State: Replica: fa748015ea2045fcbd9bc70470e0324a, State: Running, Role: LEADER
I20260927 22:07:40.853992 26021 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa748015ea2045fcbd9bc70470e0324a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35797 } }
I20260927 22:07:40.854799 26011 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:40.857965 26023 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa748015ea2045fcbd9bc70470e0324a. Latest consensus state: current_term: 1 leader_uuid: "fa748015ea2045fcbd9bc70470e0324a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa748015ea2045fcbd9bc70470e0324a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35797 } } }
I20260927 22:07:40.858702 26023 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:40.860416 26030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:40.860186 26022 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa748015ea2045fcbd9bc70470e0324a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa748015ea2045fcbd9bc70470e0324a" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 35797 } } }
I20260927 22:07:40.860695 26022 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:40.868778 26030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:40.889104 26030 catalog_manager.cc:1347] Generated new cluster ID: cd7f5819cc324862b0efaca9b0e4c554
I20260927 22:07:40.889278 26030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:40.917455 26030 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:40.919302 26030 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:40.939904 26030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa748015ea2045fcbd9bc70470e0324a: Generated new TSK 0
I20260927 22:07:40.941728 26030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:40.961238 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25944
W20260927 22:07:40.963205 26006 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43151
W20260927 22:07:40.964316 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 25944: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:40.964468 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 25944
2026-09-27T22:07:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:41 dist-test-slave-sg4j krb5kdc[26042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:41 dist-test-slave-sg4j krb5kdc[26042](info): set up 2 sockets
Sep 27 22:07:41 dist-test-slave-sg4j krb5kdc[26042](info): commencing operation
krb5kdc: starting...
W20260927 22:07:42.142839 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:42 dist-test-slave-sg4j krb5kdc[26042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:07:42.478293 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:44703
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:45287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:44703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:42.926465 26055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:42.926867 26055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:42.927043 26055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:42.935678 26055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:42.935868 26055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:42.935921 26055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:42.935984 26055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:42.936062 26055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:42.948184 26055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:45287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:44703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:44703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:42.950093 26055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:42.951859 26055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:42.968132 26061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:42.968231 26063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:42.968250 26060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:42.969472 26055 server_base.cc:1034] running on GCE node
I20260927 22:07:42.970614 26055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:42.972642 26055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:42.976115 26055 hybrid_clock.cc:648] HybridClock initialized: now 1790546862975966 us; error 84 us; skew 500 ppm
I20260927 22:07:42.976929 26055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:42 dist-test-slave-sg4j krb5kdc[26042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546862, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:42.989024 26055 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:07:42.991254 26055 webserver.cc:460] Webserver started at http://127.3.159.62:38927/ using document root <none> and password file <none>
I20260927 22:07:42.992141 26055 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:42.992306 26055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:42.992707 26055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:42.996472 26055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "6800716b11b74a3392db6c8fd4bcb3f2"
format_stamp: "Formatted at 2026-09-27 22:07:42 on dist-test-slave-sg4j"
server_key: "6eee5727f52834cf8823d0d8ca58f236"
server_key_iv: "32806831e60d138fece0481790e0f508"
server_key_version: "encryptionkey@0"
I20260927 22:07:42.997589 26055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "6800716b11b74a3392db6c8fd4bcb3f2"
format_stamp: "Formatted at 2026-09-27 22:07:42 on dist-test-slave-sg4j"
server_key: "6eee5727f52834cf8823d0d8ca58f236"
server_key_iv: "32806831e60d138fece0481790e0f508"
server_key_version: "encryptionkey@0"
I20260927 22:07:43.006112 26055 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260927 22:07:43.011623 26070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:43.013959 26055 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260927 22:07:43.014235 26055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "6800716b11b74a3392db6c8fd4bcb3f2"
format_stamp: "Formatted at 2026-09-27 22:07:42 on dist-test-slave-sg4j"
server_key: "6eee5727f52834cf8823d0d8ca58f236"
server_key_iv: "32806831e60d138fece0481790e0f508"
server_key_version: "encryptionkey@0"
I20260927 22:07:43.014524 26055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:43.021812 26055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:43.027030 26055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:43.027398 26055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:43.051412 26055 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:44703
I20260927 22:07:43.051518 26121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:44703 every 8 connection(s)
I20260927 22:07:43.053063 26055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:43.055711 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 26055
I20260927 22:07:43.055979 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:43.056456 3708 external_mini_cluster.cc:1428] Setting key 44c47d0ddf021ee5a209faf2e072d81c
I20260927 22:07:43.059088 26122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:43.067729 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:43.058788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33607 (local address 127.3.159.62:44703)
0927 22:07:43.059835 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:43.059849 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:43.059903 (+ 54us) server_negotiation.cc:407] Connection header received
0927 22:07:43.061938 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:43.061970 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:43.062438 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:43.062856 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:43.064485 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:43.064493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:43.064509 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:43.064542 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:43.064641 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:43.064666 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:43.064927 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:43.065184 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 22:07:43.065574 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":521,"threads_started":1}
I20260927 22:07:43.074532 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:43.078346 26122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2: Bootstrap starting.
I20260927 22:07:43.084416 26122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:43.086066 26125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:43.082082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46803 (local address 127.3.159.62:44703)
0927 22:07:43.082355 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:43.082363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:43.083134 (+ 771us) server_negotiation.cc:407] Connection header received
0927 22:07:43.083350 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:43.083357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:43.083445 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:43.083600 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:43.084984 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:43.084997 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:43.085004 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 22:07:43.085038 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:43.085135 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 22:07:43.085164 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:43.085323 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:43.085598 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 22:07:43.085683 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 22:07:43.086733 26122 log.cc:824] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:43.092031 26122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2: No bootstrap required, opened a new log
I20260927 22:07:43.103591 26122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6800716b11b74a3392db6c8fd4bcb3f2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44703 } }
I20260927 22:07:43.104600 26122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 22:07:43.104808 26122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6800716b11b74a3392db6c8fd4bcb3f2, State: Initialized, Role: FOLLOWER
I20260927 22:07:43.105856 26122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6800716b11b74a3392db6c8fd4bcb3f2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44703 } }
I20260927 22:07:43.106184 26122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 22:07:43.106384 26122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 22:07:43.106654 26122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:43.110375 26122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6800716b11b74a3392db6c8fd4bcb3f2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44703 } }
I20260927 22:07:43.111304 26122 leader_election.cc:302] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6800716b11b74a3392db6c8fd4bcb3f2; no voters:
I20260927 22:07:43.112289 26122 leader_election.cc:288] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 22:07:43.113191 26132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:43.115753 26132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [term 1 LEADER]: Becoming Leader. State: Replica: 6800716b11b74a3392db6c8fd4bcb3f2, State: Running, Role: LEADER
I20260927 22:07:43.116703 26132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6800716b11b74a3392db6c8fd4bcb3f2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44703 } }
I20260927 22:07:43.117025 26122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:43.120877 26133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6800716b11b74a3392db6c8fd4bcb3f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6800716b11b74a3392db6c8fd4bcb3f2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44703 } } }
I20260927 22:07:43.122155 26133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:43.121608 26134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6800716b11b74a3392db6c8fd4bcb3f2. Latest consensus state: current_term: 1 leader_uuid: "6800716b11b74a3392db6c8fd4bcb3f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6800716b11b74a3392db6c8fd4bcb3f2" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44703 } } }
I20260927 22:07:43.122853 26134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:43.124243 26141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:43.132732 26141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:43.147586 26141 catalog_manager.cc:1347] Generated new cluster ID: 06dad7500b4b436e98fcea3bf42c7ecc
I20260927 22:07:43.147779 26141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:43.170598 26141 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:43.171459 26141 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:43.182206 26141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6800716b11b74a3392db6c8fd4bcb3f2: Generated new TSK 0
I20260927 22:07:43.184059 26141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 22:07:43.192261 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 26055
W20260927 22:07:43.194212 26117 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33607
W20260927 22:07:43.195655 3708 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 26055: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 22:07:43.195940 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 26055
2026-09-27T22:07:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2231 ms)
[----------] 18 tests from AuthTokenIssuingTest (41611 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:43 dist-test-slave-sg4j krb5kdc[26153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:43 dist-test-slave-sg4j krb5kdc[26153](info): set up 2 sockets
Sep 27 22:07:43 dist-test-slave-sg4j krb5kdc[26153](info): commencing operation
krb5kdc: starting...
W20260927 22:07:44.411273 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.012s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:44 dist-test-slave-sg4j krb5kdc[26153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:07:44.755134 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:44393
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:44191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:44393,127.3.159.61:43925,127.3.159.60:33085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:45.209697 26166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:07:45.210166 26166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:45.210340 26166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:45.210517 26166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:45.219064 26166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:45.219213 26166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:45.219316 26166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:45.219414 26166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:45.219530 26166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:45.231287 26166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:44393,127.3.159.61:43925,127.3.159.60:33085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:44393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:45.233258 26166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:45.235232 26166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:45.251044 26171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:45.251375 26172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:45.251638 26174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:45.253989 26166 server_base.cc:1034] running on GCE node
I20260927 22:07:45.255235 26166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:45.258447 26166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:45.259886 26166 hybrid_clock.cc:648] HybridClock initialized: now 1790546865259782 us; error 85 us; skew 500 ppm
I20260927 22:07:45.260457 26166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:45 dist-test-slave-sg4j krb5kdc[26153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546865, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:45.265846 26166 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:07:45.271634 26166 webserver.cc:460] Webserver started at http://127.3.159.62:37197/ using document root <none> and password file <none>
I20260927 22:07:45.272794 26166 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:45.272940 26166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:45.273371 26166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:45.277839 26166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "5d30d391a7ac413cb82d869ade6959ed"
format_stamp: "Formatted at 2026-09-27 22:07:45 on dist-test-slave-sg4j"
server_key: "3b9e616fcd19c9624b744fa23c01b2e1"
server_key_iv: "d0fa18f4f3e7777b92c0653842cf30a9"
server_key_version: "encryptionkey@0"
I20260927 22:07:45.278991 26166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "5d30d391a7ac413cb82d869ade6959ed"
format_stamp: "Formatted at 2026-09-27 22:07:45 on dist-test-slave-sg4j"
server_key: "3b9e616fcd19c9624b744fa23c01b2e1"
server_key_iv: "d0fa18f4f3e7777b92c0653842cf30a9"
server_key_version: "encryptionkey@0"
I20260927 22:07:45.286765 26166 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:07:45.292405 26181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:45.295097 26166 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260927 22:07:45.295516 26166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "5d30d391a7ac413cb82d869ade6959ed"
format_stamp: "Formatted at 2026-09-27 22:07:45 on dist-test-slave-sg4j"
server_key: "3b9e616fcd19c9624b744fa23c01b2e1"
server_key_iv: "d0fa18f4f3e7777b92c0653842cf30a9"
server_key_version: "encryptionkey@0"
I20260927 22:07:45.295842 26166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:45.350788 26166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:45.359375 26166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:45.359814 26166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:45.386660 26166 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:44393
I20260927 22:07:45.386736 26232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:44393 every 8 connection(s)
I20260927 22:07:45.388454 26166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:45.390134 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 26166
I20260927 22:07:45.390396 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:45.391240 3708 external_mini_cluster.cc:1428] Setting key 11b44b45e733e348615e6588162b98cb
I20260927 22:07:45.394727 26233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:45.407101 26233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.159.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.61@KRBTEST.COM" created.
Sep 27 22:07:45 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546865, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:07:45.447945 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.413204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.3.159.62:44393)
0927 22:07:45.414073 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:45.414081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:45.414138 (+ 57us) server_negotiation.cc:407] Connection header received
0927 22:07:45.417648 (+ 3510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:45.417656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:45.418047 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:45.418388 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:45.420820 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:45.422615 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:45.425848 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:45.426696 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:45.426938 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:45.434966 (+ 8028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:45.434983 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:45.435012 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:45.435069 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:45.440204 (+ 5135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:45.441805 (+ 1601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:45.441821 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:45.441828 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:45.442182 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:45.443036 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:45.443044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:45.443050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:45.444148 (+ 1098us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:45.444464 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:45.445544 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0927 22:07:45.446277 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":229,"server-negotiator.queue_time_us":637,"thread_start_us":432,"threads_started":1}
I20260927 22:07:45.448021 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.411760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44393 (local address 127.0.0.1:38944)
0927 22:07:45.412994 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:45.413081 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:45.416761 (+ 3680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:45.418465 (+ 1704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:45.418477 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:45.419638 (+ 1161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:45.420346 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:45.420386 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:45.422916 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:45.422926 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:45.425466 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:45.425504 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:45.427138 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:45.427149 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:45.427442 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:45.428410 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:45.428460 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:45.434522 (+ 6062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:45.440559 (+ 6037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:45.440587 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:45.440605 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:45.441524 (+ 919us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:45.442510 (+ 986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:45.442518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:45.442525 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:45.442843 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:45.444543 (+ 1700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:45.444556 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:45.444915 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 22:07:45.445232 (+ 317us) client_negotiation.cc:240] Negotiation successful
0927 22:07:45.445949 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":429,"threads_started":1}
I20260927 22:07:45.452500 26233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:07:45.454309 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.453732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:42158)
0927 22:07:45.453973 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:45.454147 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 22:07:45.454759 26183 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
I20260927 22:07:45.457016 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.456435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:42170)
0927 22:07:45.456747 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:45.456833 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 22:07:45.457280 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:43925: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab.
Entry for principal kudu/127.3.159.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab.
I20260927 22:07:45.478945 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.61:43925
--webserver_interface=127.3.159.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab
--principal=kudu/127.3.159.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:44191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:44393,127.3.159.61:43925,127.3.159.60:33085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:07:45.506840 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } attempt: 1
I20260927 22:07:45.508388 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.507767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:42172)
0927 22:07:45.508018 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:45.508139 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 22:07:45.509989 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.509492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:42176)
0927 22:07:45.509728 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:45.509818 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260927 22:07:45.510218 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:43925: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
I20260927 22:07:45.578608 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } attempt: 2
I20260927 22:07:45.580168 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.579552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:42178)
0927 22:07:45.579883 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:45.579986 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 22:07:45.582000 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.581130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:42184)
0927 22:07:45.581325 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:45.581783 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 22:07:45.582394 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:43925: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
I20260927 22:07:45.673789 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } attempt: 3
I20260927 22:07:45.675474 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.674928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:42200)
0927 22:07:45.675182 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:45.675297 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 22:07:45.676817 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.676393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:42208)
0927 22:07:45.676572 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:45.676647 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 22:07:45.677112 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:43925: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
I20260927 22:07:45.820466 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } attempt: 4
I20260927 22:07:45.821800 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.821248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:42216)
0927 22:07:45.821563 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:45.821661 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260927 22:07:45.823311 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:45.822848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:42232)
0927 22:07:45.823058 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:45.823159 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 22:07:45.823662 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:43925: Network error: Client connection negotiation failed: client connection to 127.3.159.61:43925: connect: Connection refused (error 111)
W20260927 22:07:45.928100 26240 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:07:45.928570 26240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:45.928720 26240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:45.928882 26240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:45.937256 26240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:45.937443 26240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:45.937584 26240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:45.937682 26240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:45.937786 26240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:45.949729 26240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:44393,127.3.159.61:43925,127.3.159.60:33085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab
--principal=kudu/127.3.159.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.61:43925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.159.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:45.951473 26240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:45.953143 26240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:45.968573 26248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:45.968266 26246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:45.968874 26245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:45.969928 26240 server_base.cc:1034] running on GCE node
I20260927 22:07:45.971012 26240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:45.973197 26240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:45.974606 26240 hybrid_clock.cc:648] HybridClock initialized: now 1790546865974472 us; error 92 us; skew 500 ppm
I20260927 22:07:45.975183 26240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:45 dist-test-slave-sg4j krb5kdc[26153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546865, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:45.979682 26240 init.cc:377] Logged in from keytab as kudu/127.3.159.61@KRBTEST.COM (short username kudu)
I20260927 22:07:45.982026 26240 webserver.cc:460] Webserver started at http://127.3.159.61:39411/ using document root <none> and password file <none>
I20260927 22:07:45.983022 26240 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:45.983181 26240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:45.983574 26240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:45.987905 26240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/data/instance:
uuid: "4c22dc8e6fc3441989b1e63a49f458d8"
format_stamp: "Formatted at 2026-09-27 22:07:45 on dist-test-slave-sg4j"
server_key: "c130a1fb08caa7bee08d2d524986af84"
server_key_iv: "bd085bf6489ea9a2ba68f6e2f5ed98ae"
server_key_version: "encryptionkey@0"
I20260927 22:07:45.989194 26240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/wal/instance:
uuid: "4c22dc8e6fc3441989b1e63a49f458d8"
format_stamp: "Formatted at 2026-09-27 22:07:45 on dist-test-slave-sg4j"
server_key: "c130a1fb08caa7bee08d2d524986af84"
server_key_iv: "bd085bf6489ea9a2ba68f6e2f5ed98ae"
server_key_version: "encryptionkey@0"
I20260927 22:07:45.997301 26240 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260927 22:07:46.002902 26255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:46.005282 26240 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:07:46.005725 26240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/wal
uuid: "4c22dc8e6fc3441989b1e63a49f458d8"
format_stamp: "Formatted at 2026-09-27 22:07:45 on dist-test-slave-sg4j"
server_key: "c130a1fb08caa7bee08d2d524986af84"
server_key_iv: "bd085bf6489ea9a2ba68f6e2f5ed98ae"
server_key_version: "encryptionkey@0"
I20260927 22:07:46.006084 26240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:46.040827 26240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:46.046950 26240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:46.047288 26240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:46.073650 26240 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.61:43925
I20260927 22:07:46.073671 26306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.61:43925 every 8 connection(s)
I20260927 22:07:46.076061 26240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
I20260927 22:07:46.078967 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 26240
I20260927 22:07:46.079548 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-1/wal/instance
I20260927 22:07:46.080246 3708 external_mini_cluster.cc:1428] Setting key eb1a8bd122e08d94caa7077863ac85ae
I20260927 22:07:46.083673 26307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:46.098158 26307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.159.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.60@KRBTEST.COM" created.
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546865, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:07:46.123184 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } attempt: 5
I20260927 22:07:46.130651 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.102127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.3.159.62:44393)
0927 22:07:46.102842 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:46.102847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:46.104535 (+ 1688us) server_negotiation.cc:407] Connection header received
0927 22:07:46.107024 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:46.107030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:46.107162 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:46.107296 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:46.109341 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.110270 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.112301 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.112933 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.113120 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.122251 (+ 9131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:46.122258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:46.122262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:46.122341 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:46.126690 (+ 4349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.127814 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.127822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.127829 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.127991 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.128523 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.128529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.128532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.128853 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:46.129044 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:46.129918 (+ 874us) server_negotiation.cc:299] Negotiation successful
0927 22:07:46.130197 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":365,"threads_started":1}
I20260927 22:07:46.131989 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.103114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44393 (local address 127.0.0.1:38596)
0927 22:07:46.104263 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.104326 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:46.106607 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:46.107624 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:46.107634 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:46.108341 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:46.109051 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.109071 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.110525 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.110532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.112123 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.112138 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.113416 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.113428 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.113915 (+ 487us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.114775 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:46.114803 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:46.121722 (+ 6919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:46.126874 (+ 5152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.126883 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.126910 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.127644 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.128132 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.128138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.128141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.128379 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.128996 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:46.129005 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:46.129519 (+ 514us) client_negotiation.cc:769] Sending connection context
0927 22:07:46.129833 (+ 314us) client_negotiation.cc:240] Negotiation successful
0927 22:07:46.130453 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":439,"threads_started":1}
I20260927 22:07:46.134117 26307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546865, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546865, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
I20260927 22:07:46.148808 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.124110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:43114)
0927 22:07:46.124461 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.124489 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:46.124635 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:46.126560 (+ 1925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:46.126570 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:46.127262 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:46.127603 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.127621 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.130305 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.130315 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.132296 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.132335 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.134223 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.134231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.134507 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.135266 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:46.135293 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:46.138495 (+ 3202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:46.144665 (+ 6170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.144673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.144677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.145327 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.146000 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.146005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.146009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.146197 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.147297 (+ 1100us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:46.147308 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:46.147558 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 22:07:46.147799 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 22:07:46.148226 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260927 22:07:46.149224 26312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.124590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43114 (local address 127.3.159.61:43925)
0927 22:07:46.125712 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:46.125726 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:46.125771 (+ 45us) server_negotiation.cc:407] Connection header received
0927 22:07:46.125881 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:46.125891 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:46.126230 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:46.126447 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:46.127938 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.130043 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.132625 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.133895 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.134179 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.138774 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:46.138788 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:46.138811 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:46.138852 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:46.144421 (+ 5569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.145564 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.145579 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.145585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.145802 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.146347 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.146355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.146361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.146973 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:46.147258 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:46.148085 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 22:07:46.148578 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":528,"threads_started":1}
I20260927 22:07:46.153713 26233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:07:46.155297 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.154725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50490)
0927 22:07:46.154956 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.155062 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 22:07:46.155758 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.135171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:43122)
0927 22:07:46.135448 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.135472 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:46.135600 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:46.136913 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:46.136922 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:46.137572 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:46.137829 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.137842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.139785 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.139792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.140776 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.140790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.142124 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.142133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.142375 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.143019 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:46.143053 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:46.146822 (+ 3769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:46.151813 (+ 4991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.151822 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.151829 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.152630 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.153309 (+ 679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.153318 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.153325 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.153595 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.154400 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:46.154408 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:46.154668 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 22:07:46.154866 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 22:07:46.155262 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 22:07:46.156055 26314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.135656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.3.159.61:43925)
0927 22:07:46.136492 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:46.136501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:46.136527 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:07:46.136586 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:46.136591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:46.136675 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:46.136805 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:46.138041 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.139602 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.140973 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.141894 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.142074 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.147153 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:46.147162 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:46.147170 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:46.147215 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:46.151562 (+ 4347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.152859 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.152865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.152869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.153106 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.153768 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.153773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.153777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.154222 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:46.155062 (+ 840us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:46.155227 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 22:07:46.155529 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":414,"threads_started":1}
I20260927 22:07:46.157096 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.156667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50494)
0927 22:07:46.156834 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.156928 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 22:07:46.157516 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:33085: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
I20260927 22:07:46.157747 26307 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:07:46.159526 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.158760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50502)
0927 22:07:46.159158 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.159317 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260927 22:07:46.159909 26259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
I20260927 22:07:46.162523 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.161966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50516)
0927 22:07:46.162185 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.162345 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 22:07:46.162817 26307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:33085: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab.
Entry for principal kudu/127.3.159.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab.
I20260927 22:07:46.171370 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.60:33085
--webserver_interface=127.3.159.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab
--principal=kudu/127.3.159.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:44191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:44393,127.3.159.61:43925,127.3.159.60:33085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:07:46.208869 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } attempt: 1
I20260927 22:07:46.210768 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.210018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50518)
0927 22:07:46.210322 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.210462 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 22:07:46.212409 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.211732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50532)
0927 22:07:46.212136 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.212249 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 22:07:46.212222 26307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } attempt: 1
W20260927 22:07:46.212891 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:33085: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
I20260927 22:07:46.214138 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.213526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50538)
0927 22:07:46.213786 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.213914 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 22:07:46.216111 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.215376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50548)
0927 22:07:46.215740 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.215906 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
W20260927 22:07:46.216500 26307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:33085: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
I20260927 22:07:46.281215 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } attempt: 2
I20260927 22:07:46.283138 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.282218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50554)
0927 22:07:46.282667 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.282835 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260927 22:07:46.284698 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.284127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50564)
0927 22:07:46.284368 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.284511 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 22:07:46.284865 26307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } attempt: 2
W20260927 22:07:46.285063 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:33085: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
I20260927 22:07:46.286325 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.285737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50570)
0927 22:07:46.286023 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.286141 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 22:07:46.287777 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.287219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50574)
0927 22:07:46.287387 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.287474 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 22:07:46.288033 26307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:33085: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
I20260927 22:07:46.379359 26307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } attempt: 3
I20260927 22:07:46.381125 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.380467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50580)
0927 22:07:46.380831 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.380947 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260927 22:07:46.382752 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.382143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50596)
0927 22:07:46.382501 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.382616 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
W20260927 22:07:46.382995 26307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:33085: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
I20260927 22:07:46.391347 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } attempt: 3
I20260927 22:07:46.392782 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.392230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50606)
0927 22:07:46.392471 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.392584 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260927 22:07:46.394064 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.393607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50612)
0927 22:07:46.393820 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.393945 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260927 22:07:46.394393 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:33085: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
I20260927 22:07:46.526325 26307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } attempt: 4
I20260927 22:07:46.527545 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.527090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50628)
0927 22:07:46.527293 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.527388 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:07:46.529029 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.528547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50634)
0927 22:07:46.528814 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.528896 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260927 22:07:46.529438 26307 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:33085: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
I20260927 22:07:46.571699 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } attempt: 4
I20260927 22:07:46.573282 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.572651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50644)
0927 22:07:46.572942 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.573069 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260927 22:07:46.574497 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.574134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50652)
0927 22:07:46.574293 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.574377 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 22:07:46.574725 26233 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:33085: Network error: Client connection negotiation failed: client connection to 127.3.159.60:33085: connect: Connection refused (error 111)
W20260927 22:07:46.634398 26316 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:07:46.634934 26316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:46.635075 26316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:46.635223 26316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:46.643957 26316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:46.644155 26316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:46.644273 26316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:46.644379 26316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:46.644507 26316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:46.657207 26316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:44191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:44393,127.3.159.61:43925,127.3.159.60:33085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab
--principal=kudu/127.3.159.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.60:33085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.159.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:46.659482 26316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:46.661825 26316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:46.679112 26322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:46.679235 26321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:46.679198 26324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:46.680408 26316 server_base.cc:1034] running on GCE node
I20260927 22:07:46.681507 26316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:46.683429 26316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:46.684828 26316 hybrid_clock.cc:648] HybridClock initialized: now 1790546866684734 us; error 57 us; skew 500 ppm
I20260927 22:07:46.685468 26316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546866, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:46.690054 26316 init.cc:377] Logged in from keytab as kudu/127.3.159.60@KRBTEST.COM (short username kudu)
I20260927 22:07:46.692404 26316 webserver.cc:460] Webserver started at http://127.3.159.60:36343/ using document root <none> and password file <none>
I20260927 22:07:46.693344 26316 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:46.693552 26316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:46.693924 26316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:46.697363 26316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/data/instance:
uuid: "4b4939f2c98b46f48454779ce2175c6b"
format_stamp: "Formatted at 2026-09-27 22:07:46 on dist-test-slave-sg4j"
server_key: "df96ba1e919fc83df0369230bfb2e843"
server_key_iv: "3bc540d81402c1f7fa3419a215e2bfcd"
server_key_version: "encryptionkey@0"
I20260927 22:07:46.698443 26316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/wal/instance:
uuid: "4b4939f2c98b46f48454779ce2175c6b"
format_stamp: "Formatted at 2026-09-27 22:07:46 on dist-test-slave-sg4j"
server_key: "df96ba1e919fc83df0369230bfb2e843"
server_key_iv: "3bc540d81402c1f7fa3419a215e2bfcd"
server_key_version: "encryptionkey@0"
I20260927 22:07:46.706408 26316 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260927 22:07:46.711426 26331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:46.713884 26316 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260927 22:07:46.714241 26316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/wal
uuid: "4b4939f2c98b46f48454779ce2175c6b"
format_stamp: "Formatted at 2026-09-27 22:07:46 on dist-test-slave-sg4j"
server_key: "df96ba1e919fc83df0369230bfb2e843"
server_key_iv: "3bc540d81402c1f7fa3419a215e2bfcd"
server_key_version: "encryptionkey@0"
I20260927 22:07:46.714577 26316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:46.756098 26316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:46.763823 26316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:46.764153 26316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:46.792675 26316 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.60:33085
I20260927 22:07:46.792788 26382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.60:33085 every 8 connection(s)
I20260927 22:07:46.795637 26316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
I20260927 22:07:46.802052 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 26316
I20260927 22:07:46.802357 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790546447244210-3708-0/minicluster-data/master-2/wal/instance
I20260927 22:07:46.803081 3708 external_mini_cluster.cc:1428] Setting key f5bc9034bbb5e217da1cb81a9598c269
I20260927 22:07:46.803450 26383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:46.817451 26383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
I20260927 22:07:46.828732 26307 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } attempt: 5
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546866, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:07:46.842226 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.806196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.3.159.60:33085)
0927 22:07:46.807466 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:46.807482 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:46.807547 (+ 65us) server_negotiation.cc:407] Connection header received
0927 22:07:46.810431 (+ 2884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:46.810468 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:46.811108 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:46.811713 (+ 605us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:46.814216 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.815768 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.818696 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.819842 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.820209 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.827667 (+ 7458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:46.827682 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:46.827702 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:46.827768 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:46.834277 (+ 6509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.835814 (+ 1537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.835828 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.835844 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.836280 (+ 436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.837157 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.837165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.837169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.838355 (+ 1186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:46.838832 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:46.839499 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 22:07:46.840105 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":596,"threads_started":1}
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546865, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
I20260927 22:07:46.853446 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.822637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:44393 (local address 127.0.0.1:38602)
0927 22:07:46.824269 (+ 1632us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.824350 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:46.824679 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:46.825651 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:46.825657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:46.826586 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:46.827641 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.827663 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.830627 (+ 2964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.830637 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.834816 (+ 4179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.834835 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.836218 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.836227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.836459 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.837865 (+ 1406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:46.837904 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:46.841609 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:46.847537 (+ 5928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.847563 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.847573 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.849295 (+ 1722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.850420 (+ 1125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.850432 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.850440 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.850740 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.851754 (+ 1014us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:46.851767 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:46.852096 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 22:07:46.852349 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 22:07:46.852807 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"thread_start_us":769,"threads_started":1}
I20260927 22:07:46.853806 26387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.822664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.3.159.62:44393)
0927 22:07:46.823501 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:46.823509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:46.824608 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 22:07:46.824987 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:46.824994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:46.825212 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:46.825504 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:46.828312 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.830178 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.835290 (+ 5112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.835953 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.836133 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.841984 (+ 5851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:46.841994 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:46.842001 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:46.842061 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:46.847245 (+ 5184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.849882 (+ 2637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.849890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.849895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.850182 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.850948 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.850955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.850957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.851530 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:46.851742 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:46.852898 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0927 22:07:46.853223 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":366,"threads_started":1}
I20260927 22:07:46.854064 26233 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } attempt: 5
I20260927 22:07:46.854519 26311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.829930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50674)
0927 22:07:46.830296 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.830319 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:46.830576 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:46.833310 (+ 2734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:46.833318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:46.833995 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:46.834271 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.834287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.836800 (+ 2513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.836809 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.838235 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.838262 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.839948 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.839962 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.840221 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.841102 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:46.841128 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:46.844424 (+ 3296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:46.851170 (+ 6746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.851177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.851181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.851899 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.852391 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.852397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.852401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.852590 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.853480 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:46.853486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:46.853693 (+ 207us) client_negotiation.cc:769] Sending connection context
0927 22:07:46.853864 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 22:07:46.854233 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260927 22:07:46.855886 26389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.830070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.3.159.60:33085)
0927 22:07:46.832256 (+ 2186us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:46.832264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:46.832313 (+ 49us) server_negotiation.cc:407] Connection header received
0927 22:07:46.832408 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:46.832415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:46.832926 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:46.833134 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:46.834573 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.836581 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.838526 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.839416 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.840642 (+ 1226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.844985 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:46.844997 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:46.845006 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:46.845078 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:46.850918 (+ 5840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.852067 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.852073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.852079 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.852254 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.852719 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.852725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.852729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.853169 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:46.853342 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:46.855124 (+ 1782us) server_negotiation.cc:299] Negotiation successful
0927 22:07:46.855414 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1977,"thread_start_us":1108,"threads_started":1}
I20260927 22:07:46.856848 26383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:07:46.863643 26307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8: Bootstrap starting.
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546865, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
I20260927 22:07:46.870550 26307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:46.872624 26307 log.cc:824] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:46.875519 26236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.854760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50684)
0927 22:07:46.854960 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.854979 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:46.855082 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:46.855469 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:46.855475 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:46.856067 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:46.856463 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.856474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.858064 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.858070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.859588 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.859605 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.860740 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.860747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.860979 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.861766 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:46.861799 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:46.865764 (+ 3965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:46.871353 (+ 5589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.871362 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.871366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.872003 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.872644 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.872652 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.872658 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.872869 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.873845 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:46.873853 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:46.874099 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 22:07:46.874280 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 22:07:46.874817 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":79}
I20260927 22:07:46.877779 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.848336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.3.159.62:44393)
0927 22:07:46.851397 (+ 3061us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:46.851402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:46.851430 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:07:46.851483 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:46.851487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:46.851576 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:46.851729 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:46.856080 (+ 4351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.857518 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.859197 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.859776 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.859935 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.865977 (+ 6042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:46.865989 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:46.865998 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:46.866061 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:46.871234 (+ 5173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.872422 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.872428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.872432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.872619 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.873160 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.873166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.873171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.873597 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:46.876605 (+ 3008us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:46.876830 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 22:07:46.877193 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":2425,"thread_start_us":2141,"threads_started":1}
I20260927 22:07:46.877920 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.854908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50684 (local address 127.3.159.60:33085)
0927 22:07:46.855100 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:46.855106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:46.855129 (+ 23us) server_negotiation.cc:407] Connection header received
0927 22:07:46.855184 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:46.855188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:46.855264 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:46.856203 (+ 939us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:46.856678 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.857881 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.859838 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.860525 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.861349 (+ 824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.866137 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:46.866145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:46.866149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:46.866202 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:46.871013 (+ 4811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.872238 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.872243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.872247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.872449 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.873044 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.873049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.873052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.873637 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:46.876293 (+ 2656us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:46.876534 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 22:07:46.877294 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546866, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
I20260927 22:07:46.885035 26233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed: Bootstrap starting.
I20260927 22:07:46.885119 26307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8: No bootstrap required, opened a new log
I20260927 22:07:46.889091 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.857892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:43925 (local address 127.0.0.1:43128)
0927 22:07:46.858329 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.858350 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:46.858459 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:46.867002 (+ 8543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:46.867010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:46.867717 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:46.868049 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.868075 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.871553 (+ 3478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.871559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.873104 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.873132 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.874660 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.874668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.874940 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.875866 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:46.875893 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:46.880880 (+ 4987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:46.885855 (+ 4975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.885862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.885865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.886474 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.887012 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.887017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.887021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.887199 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.888026 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:46.888034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:46.888229 (+ 195us) client_negotiation.cc:769] Sending connection context
0927 22:07:46.888378 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 22:07:46.888716 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306}
I20260927 22:07:46.890489 26391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.865616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43128 (local address 127.3.159.61:43925)
0927 22:07:46.866457 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:46.866467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:46.866494 (+ 27us) server_negotiation.cc:407] Connection header received
0927 22:07:46.866578 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:46.866586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:46.866688 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:46.868293 (+ 1605us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:46.868549 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.871374 (+ 2825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.873352 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.874310 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.875560 (+ 1250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.881272 (+ 5712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:46.881279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:46.881288 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:46.881329 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:46.885637 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.886671 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.886677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.886680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.886869 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.887365 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.887370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.887374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.887878 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:46.889583 (+ 1705us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:46.889748 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 22:07:46.890022 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":390,"threads_started":1}
I20260927 22:07:46.890929 26233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:46.891733 26383 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:07:46.892583 26233 log.cc:824] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:46.902701 26307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } }
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546866, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
I20260927 22:07:46.903292 26307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c22dc8e6fc3441989b1e63a49f458d8, State: Initialized, Role: FOLLOWER
I20260927 22:07:46.904038 26307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } }
Sep 27 22:07:46 dist-test-slave-sg4j krb5kdc[26153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
I20260927 22:07:46.910765 26395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } } }
I20260927 22:07:46.911537 26395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:46.912922 26307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:46.914639 26388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.892521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:33085 (local address 127.0.0.1:50686)
0927 22:07:46.893308 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:46.893333 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:46.893544 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:46.894116 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:46.894124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:46.894811 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:46.895122 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.895140 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.897015 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.897024 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.898791 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:46.898809 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.900063 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.900073 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:46.900333 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.901158 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:46.901181 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:46.904123 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:46.909144 (+ 5021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.909149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.909153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.911345 (+ 2192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.911985 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:46.911991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:46.911995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:46.912155 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:46.913326 (+ 1171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:46.913332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:46.913544 (+ 212us) client_negotiation.cc:769] Sending connection context
0927 22:07:46.913717 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 22:07:46.914146 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674}
I20260927 22:07:46.916293 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.892973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.3.159.60:33085)
0927 22:07:46.893171 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:46.893179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:46.893426 (+ 247us) server_negotiation.cc:407] Connection header received
0927 22:07:46.893718 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:46.893726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:46.893815 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:46.893912 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:46.895322 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.896814 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.899031 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.899799 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.899981 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.904344 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:46.904353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:46.904359 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:46.904399 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:46.908948 (+ 4549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.911539 (+ 2591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.911544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.911547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.911809 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.912278 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.912282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.912285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.912725 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:46.913250 (+ 525us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:46.914078 (+ 828us) server_negotiation.cc:299] Negotiation successful
0927 22:07:46.914466 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 22:07:46.925916 26383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b: Bootstrap starting.
I20260927 22:07:46.927843 26233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed: No bootstrap required, opened a new log
I20260927 22:07:46.928494 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:46.880740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43138 (local address 127.3.159.61:43925)
0927 22:07:46.897962 (+ 17222us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:46.897968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:46.898001 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:07:46.898068 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:46.898072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:46.898153 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:46.898302 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:46.900280 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.901626 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.903819 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:46.905160 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:46.905307 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:46.912747 (+ 7440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:46.912757 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:46.914654 (+ 1897us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:46.915053 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:46.920936 (+ 5883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.922602 (+ 1666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.922616 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.922622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.923008 (+ 386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:46.923857 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:46.923866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:46.923871 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:46.924479 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:46.924912 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:46.927014 (+ 2102us) server_negotiation.cc:299] Negotiation successful
0927 22:07:46.927457 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17053,"thread_start_us":16886,"threads_started":1}
I20260927 22:07:46.931182 26383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:46.933928 26383 log.cc:824] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:46.939234 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:46.943105 26383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b: No bootstrap required, opened a new log
I20260927 22:07:46.945995 26208 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5d30d391a7ac413cb82d869ade6959ed"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38618
I20260927 22:07:46.946058 26233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } }
I20260927 22:07:46.947723 26233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d30d391a7ac413cb82d869ade6959ed, State: Initialized, Role: FOLLOWER
I20260927 22:07:46.949024 26233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } }
I20260927 22:07:46.952724 26401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } } }
I20260927 22:07:46.956408 26401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:46.956961 26233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:46.957293 26208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:07:46.957769 26208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:46.965060 26208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } }
I20260927 22:07:46.965134 26383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } }
I20260927 22:07:46.966058 26383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4939f2c98b46f48454779ce2175c6b, State: Initialized, Role: FOLLOWER
I20260927 22:07:46.967371 26383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } }
I20260927 22:07:46.971297 26282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d30d391a7ac413cb82d869ade6959ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4c22dc8e6fc3441989b1e63a49f458d8"
I20260927 22:07:46.972470 26282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:46.972455 26405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } } }
I20260927 22:07:46.973105 26405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:46.975646 26383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:46.979753 26282 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d30d391a7ac413cb82d869ade6959ed in term 1.
I20260927 22:07:46.982975 26358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5d30d391a7ac413cb82d869ade6959ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4b4939f2c98b46f48454779ce2175c6b"
I20260927 22:07:46.983889 26358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:46.990075 26358 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d30d391a7ac413cb82d869ade6959ed in term 1.
I20260927 22:07:46.991564 26185 leader_election.cc:302] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b4939f2c98b46f48454779ce2175c6b, 5d30d391a7ac413cb82d869ade6959ed; no voters:
I20260927 22:07:46.992655 26401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:46.994483 26208 leader_election.cc:288] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [CANDIDATE]: Term 1 election: Requested vote from peers 4c22dc8e6fc3441989b1e63a49f458d8 (127.3.159.61:43925), 4b4939f2c98b46f48454779ce2175c6b (127.3.159.60:33085)
I20260927 22:07:47.010878 26401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [term 1 LEADER]: Becoming Leader. State: Replica: 5d30d391a7ac413cb82d869ade6959ed, State: Running, Role: LEADER
I20260927 22:07:47.011982 26401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } }
W20260927 22:07:47.017601 26337 tablet.cc:2369] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:07:47.024334 26431 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:07:47.031709 26418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d30d391a7ac413cb82d869ade6959ed. Latest consensus state: current_term: 1 leader_uuid: "5d30d391a7ac413cb82d869ade6959ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } } }
I20260927 22:07:47.032351 26418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [sys.catalog]: This master's current role is: LEADER
W20260927 22:07:47.051175 26261 tablet.cc:2369] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:07:47.030109 26240 master_runner.cc:186] Error getting master registration for 127.3.159.62:44393: OK, instance_id { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" instance_seqno: 1790546865379649 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:07:47.051972 26240 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:07:47.052357 26440 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:07:47.057008 26437 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:07:47.055383 26316 master_runner.cc:186] Error getting master registration for 127.3.159.62:44393: OK, instance_id { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" instance_seqno: 1790546865379649 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:07:47.067708 26316 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:07:47.058337 26436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:47.084915 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:47.035034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.3.159.61:43925)
0927 22:07:47.035343 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.035358 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.053929 (+ 18571us) server_negotiation.cc:407] Connection header received
0927 22:07:47.054707 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.054713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.054827 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.054918 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.059820 (+ 4902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:47.061024 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:47.065669 (+ 4645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:47.066574 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:47.066713 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:47.070770 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:47.070782 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:47.070789 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:47.070859 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:47.076116 (+ 5257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:47.077187 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:47.077196 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:47.077201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:47.077417 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:47.082534 (+ 5117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:47.082541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:47.082546 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:47.083112 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:47.083467 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:47.084159 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 22:07:47.084473 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 22:07:47.086891 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:47.025276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.3.159.62:44393)
0927 22:07:47.025631 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.025640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.048275 (+ 22635us) server_negotiation.cc:407] Connection header received
0927 22:07:47.049331 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.049339 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.049506 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.049622 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.057607 (+ 7985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:47.058931 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:47.064241 (+ 5310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:47.065069 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:47.065184 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:47.076631 (+ 11447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:47.076639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:47.076645 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:47.076707 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:47.082156 (+ 5449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:47.083383 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:47.083391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:47.083395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:47.083591 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:47.084062 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:47.084067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:47.084070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:47.084461 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:47.084752 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:47.086178 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0927 22:07:47.086502 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260927 22:07:47.087567 26436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 22:07:47.092509 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:47.046247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50692 (local address 127.3.159.60:33085)
0927 22:07:47.046565 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.046576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.054714 (+ 8138us) server_negotiation.cc:407] Connection header received
0927 22:07:47.055157 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.055163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.055261 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.055347 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.057707 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:47.058974 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:47.063737 (+ 4763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:47.064686 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:47.064830 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:47.078122 (+ 13292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:47.078132 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:47.078138 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:47.078204 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:47.082255 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:47.089201 (+ 6946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:47.089209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:47.089214 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:47.089547 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:47.090065 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:47.090074 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:47.090077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:47.090516 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:47.090821 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:47.091657 (+ 836us) server_negotiation.cc:299] Negotiation successful
0927 22:07:47.092070 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 22:07:47.104947 26358 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [term 1 FOLLOWER]: Refusing update from remote peer 5d30d391a7ac413cb82d869ade6959ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:07:47.106483 26401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:07:47.107085 26282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [term 1 FOLLOWER]: Refusing update from remote peer 5d30d391a7ac413cb82d869ade6959ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:07:47.108202 26401 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 22:07:47.114779 26187 tablet.cc:2369] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:07:47.133881 26395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d30d391a7ac413cb82d869ade6959ed. Latest consensus state: current_term: 1 leader_uuid: "5d30d391a7ac413cb82d869ade6959ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } } }
I20260927 22:07:47.134971 26395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:47.135898 26405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d30d391a7ac413cb82d869ade6959ed. Latest consensus state: current_term: 1 leader_uuid: "5d30d391a7ac413cb82d869ade6959ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } } }
I20260927 22:07:47.136296 26405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:47.139420 26443 mvcc.cc:205] Tried to move back new op lower bound from 7334079967645196288 to 7334079967307964416. Current Snapshot: MvccSnapshot[applied={T|T < 7334079967645196288}]
I20260927 22:07:47.149159 26418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d30d391a7ac413cb82d869ade6959ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } } }
I20260927 22:07:47.149817 26418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:47.150825 26401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5d30d391a7ac413cb82d869ade6959ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } } }
I20260927 22:07:47.151369 26401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:47.151294 26395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d30d391a7ac413cb82d869ade6959ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } } }
I20260927 22:07:47.151960 26395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:47.153635 26436 catalog_manager.cc:1347] Generated new cluster ID: c65e6d5d324f4c5e8e86ace59e0f9ee5
I20260927 22:07:47.153868 26436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:47.154637 26405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5d30d391a7ac413cb82d869ade6959ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d30d391a7ac413cb82d869ade6959ed" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 44393 } } peers { permanent_uuid: "4c22dc8e6fc3441989b1e63a49f458d8" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 43925 } } peers { permanent_uuid: "4b4939f2c98b46f48454779ce2175c6b" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 33085 } } }
I20260927 22:07:47.154968 26405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:47.189555 26436 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:47.192904 26436 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:47.215492 26436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d30d391a7ac413cb82d869ade6959ed: Generated new TSK 0
I20260927 22:07:47.217619 26436 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:07:47.335858 26390 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.324004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.3.159.62:44393)
0927 22:07:47.324269 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.324279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.331717 (+ 7438us) server_negotiation.cc:407] Connection header received
0927 22:07:47.332621 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.332633 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.332833 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.332949 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.335406 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38634: BlockingRecv error: recv got EOF from 127.0.0.1:38634 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 22:07:47.338093 26386 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.323122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.3.159.60:33085)
0927 22:07:47.323653 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.323663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.335436 (+ 11773us) server_negotiation.cc:407] Connection header received
0927 22:07:47.335770 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.335777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.335879 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.335962 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.337769 (+ 1807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50696: BlockingRecv error: recv got EOF from 127.0.0.1:50696 (error 108)
Metrics: {"server-negotiator.queue_time_us":336}
W20260927 22:07:47.338670 26393 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.323849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43144 (local address 127.3.159.61:43925)
0927 22:07:47.324061 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.324068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.335235 (+ 11167us) server_negotiation.cc:407] Connection header received
0927 22:07:47.335624 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.335630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.335744 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.335843 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.338156 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43144: BlockingRecv error: recv got EOF from 127.0.0.1:43144 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 22:07:47.349676 26393 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.346475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.3.159.61:43925)
0927 22:07:47.346763 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.346772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.347251 (+ 479us) server_negotiation.cc:407] Connection header received
0927 22:07:47.347684 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.347690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.347879 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.348043 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.349242 (+ 1199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43148: BlockingRecv error: recv got EOF from 127.0.0.1:43148 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
W20260927 22:07:47.360707 26393 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.357607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.3.159.61:43925)
0927 22:07:47.357897 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.357905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.358227 (+ 322us) server_negotiation.cc:407] Connection header received
0927 22:07:47.358601 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.358608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.358963 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.359148 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.360408 (+ 1260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43160: BlockingRecv error: recv got EOF from 127.0.0.1:43160 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
W20260927 22:07:47.371776 26393 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.368670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43172 (local address 127.3.159.61:43925)
0927 22:07:47.368998 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.369006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.369577 (+ 571us) server_negotiation.cc:407] Connection header received
0927 22:07:47.369900 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.369907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.370022 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.370165 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.371410 (+ 1245us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43172: BlockingRecv error: recv got EOF from 127.0.0.1:43172 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
W20260927 22:07:47.384389 26393 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.380958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.3.159.61:43925)
0927 22:07:47.381351 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.381359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.382210 (+ 851us) server_negotiation.cc:407] Connection header received
0927 22:07:47.382591 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.382596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.382702 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.382909 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.384040 (+ 1131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43184: BlockingRecv error: recv got EOF from 127.0.0.1:43184 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
W20260927 22:07:47.401023 26393 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.398168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43194 (local address 127.3.159.61:43925)
0927 22:07:47.398401 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.398408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.399115 (+ 707us) server_negotiation.cc:407] Connection header received
0927 22:07:47.399404 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.399421 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.399537 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.399619 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.400772 (+ 1153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43194: BlockingRecv error: recv got EOF from 127.0.0.1:43194 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 22:07:47.425706 26393 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.423020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.3.159.61:43925)
0927 22:07:47.423361 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.423370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.423854 (+ 484us) server_negotiation.cc:407] Connection header received
0927 22:07:47.424066 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.424072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.424183 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.424345 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.425460 (+ 1115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43200: BlockingRecv error: recv got EOF from 127.0.0.1:43200 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
W20260927 22:07:47.465929 26393 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.463073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.3.159.61:43925)
0927 22:07:47.463395 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.463403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.463884 (+ 481us) server_negotiation.cc:407] Connection header received
0927 22:07:47.464190 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.464195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.464291 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.464398 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.465626 (+ 1228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43216: BlockingRecv error: recv got EOF from 127.0.0.1:43216 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
W20260927 22:07:47.539155 26393 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.536302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.3.159.61:43925)
0927 22:07:47.536514 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.536520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.537017 (+ 497us) server_negotiation.cc:407] Connection header received
0927 22:07:47.537303 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.537309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.537530 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.537658 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.538845 (+ 1187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43220: BlockingRecv error: recv got EOF from 127.0.0.1:43220 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 22:07:47.674768 26393 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.672291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.3.159.61:43925)
0927 22:07:47.672587 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.672595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.673022 (+ 427us) server_negotiation.cc:407] Connection header received
0927 22:07:47.673215 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.673220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.673349 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.673560 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.674473 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43224: BlockingRecv error: recv got EOF from 127.0.0.1:43224 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260927 22:07:47.938783 26393 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:47.936548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.3.159.61:43925)
0927 22:07:47.936802 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:47.936809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:47.936835 (+ 26us) server_negotiation.cc:407] Connection header received
0927 22:07:47.937036 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:47.937041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:47.937163 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:47.937310 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:47.938312 (+ 1002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43226: BlockingRecv error: recv got EOF from 127.0.0.1:43226 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 22:07:48.030622 26431 catalog_manager.cc:1259] Loaded cluster ID: c65e6d5d324f4c5e8e86ace59e0f9ee5
I20260927 22:07:48.030874 26431 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8: loading cluster ID for follower catalog manager: success
I20260927 22:07:48.034201 26431 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8: acquiring CA information for follower catalog manager: success
I20260927 22:07:48.037920 26431 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c22dc8e6fc3441989b1e63a49f458d8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:07:48.060701 26440 catalog_manager.cc:1259] Loaded cluster ID: c65e6d5d324f4c5e8e86ace59e0f9ee5
I20260927 22:07:48.060896 26440 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b: loading cluster ID for follower catalog manager: success
I20260927 22:07:48.066375 26440 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b: acquiring CA information for follower catalog manager: success
I20260927 22:07:48.070307 26440 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b4939f2c98b46f48454779ce2175c6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:07:48.483992 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:48.459630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.3.159.61:43925)
0927 22:07:48.473008 (+ 13378us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:48.473015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:48.473048 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:07:48.473149 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:48.473155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:48.473407 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:48.473825 (+ 418us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:07:48.474570 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:48.476421 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:48.478800 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:48.479634 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:48.479839 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:48.480763 (+ 924us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:07:48.482236 (+ 1473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:07:48.482481 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:48.482853 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 22:07:48.483517 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13215,"thread_start_us":12417,"threads_started":1}
I20260927 22:07:48.515498 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:48.501801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.3.159.60:33085)
0927 22:07:48.503119 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:48.503129 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:48.503248 (+ 119us) server_negotiation.cc:407] Connection header received
0927 22:07:48.503449 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:48.503456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:48.503783 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:48.504062 (+ 279us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:07:48.505330 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:48.507364 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:48.510414 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:48.511343 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:48.511594 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:48.512875 (+ 1281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:07:48.514288 (+ 1413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:07:48.514643 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:48.514776 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 22:07:48.514902 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":819,"threads_started":1}
I20260927 22:07:48.544420 26518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:48.528765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.3.159.61:43925)
0927 22:07:48.529084 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:48.529095 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:48.536012 (+ 6917us) server_negotiation.cc:407] Connection header received
0927 22:07:48.536949 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:48.536960 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:48.537116 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:48.537211 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:07:48.539285 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:48.540146 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:48.542159 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:48.542724 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:48.542907 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:48.543462 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:07:48.543604 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:07:48.543713 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:48.543885 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 22:07:48.543992 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260927 22:07:48.552661 26530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:48.528168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.3.159.62:44393)
0927 22:07:48.544436 (+ 16268us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:48.544448 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:48.544489 (+ 41us) server_negotiation.cc:407] Connection header received
0927 22:07:48.544676 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:48.544685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:48.545008 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:48.545367 (+ 359us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:07:48.546245 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:48.547406 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:48.549569 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:48.550061 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:48.550235 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:48.550922 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:07:48.551575 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:07:48.551687 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:48.552035 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 22:07:48.552155 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16033,"thread_start_us":17428,"threads_started":1}
I20260927 22:07:48.554404 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:48.528308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.3.159.60:33085)
0927 22:07:48.529217 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:48.529227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:48.544442 (+ 15215us) server_negotiation.cc:407] Connection header received
0927 22:07:48.544820 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:48.544828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:48.544948 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:48.545026 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:07:48.546006 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:48.547301 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:48.550902 (+ 3601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:48.551550 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:48.551711 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:48.553249 (+ 1538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:07:48.553485 (+ 236us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:07:48.553610 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:48.553847 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 22:07:48.553964 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 22:07:48.561419 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 26166
I20260927 22:07:48.563110 26228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:48.702258 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 26166
W20260927 22:07:48.729773 26258 connection.cc:535] server connection from 127.0.0.1:43114 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 22:07:48.730293 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 26240
I20260927 22:07:48.732074 26302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:48.880671 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 26240
I20260927 22:07:48.910104 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 26316
I20260927 22:07:48.912884 26378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:49.038443 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 26316
2026-09-27T22:07:49Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5847 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 22:07:49 dist-test-slave-sg4j krb5kdc[26538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 22:07:49 dist-test-slave-sg4j krb5kdc[26538](info): set up 2 sockets
Sep 27 22:07:49 dist-test-slave-sg4j krb5kdc[26538](info): commencing operation
krb5kdc: starting...
W20260927 22:07:50.254597 3708 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.006s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/test-user.keytab.
Sep 27 22:07:50 dist-test-slave-sg4j krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T22:07:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T22:07:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.159.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
Entry for principal kudu/127.3.159.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab.
I20260927 22:07:50.644770 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37619
--webserver_interface=127.3.159.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:37619,127.3.159.61:45999,127.3.159.60:38013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 22:07:51.087548 26551 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:07:51.088007 26551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:51.088148 26551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:51.088291 26551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:51.097157 26551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:51.097361 26551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:51.097556 26551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:51.097648 26551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:51.097751 26551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:51.109704 26551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:37619,127.3.159.61:45999,127.3.159.60:38013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.62.keytab
--principal=kudu/127.3.159.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.62:37619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.159.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:51.111608 26551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:51.113503 26551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:51.128938 26556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:51.128938 26557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:51.128953 26559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:51.129817 26551 server_base.cc:1034] running on GCE node
I20260927 22:07:51.131191 26551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:51.134030 26551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:51.135406 26551 hybrid_clock.cc:648] HybridClock initialized: now 1790546871135293 us; error 75 us; skew 500 ppm
I20260927 22:07:51.135878 26551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:51 dist-test-slave-sg4j krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546871, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:51.139755 26551 init.cc:377] Logged in from keytab as kudu/127.3.159.62@KRBTEST.COM (short username kudu)
I20260927 22:07:51.141820 26551 webserver.cc:460] Webserver started at http://127.3.159.62:43359/ using document root <none> and password file <none>
I20260927 22:07:51.142700 26551 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:51.142805 26551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:51.143107 26551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:51.146569 26551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/data/instance:
uuid: "87b4d93d107e4993ac8c1b12150307e3"
format_stamp: "Formatted at 2026-09-27 22:07:51 on dist-test-slave-sg4j"
server_key: "49a7d69b1455cc8cc745bebd7ef2087c"
server_key_iv: "4dd9e99cb0be41d0c623db6e0e11e635"
server_key_version: "encryptionkey@0"
I20260927 22:07:51.147425 26551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/wal/instance:
uuid: "87b4d93d107e4993ac8c1b12150307e3"
format_stamp: "Formatted at 2026-09-27 22:07:51 on dist-test-slave-sg4j"
server_key: "49a7d69b1455cc8cc745bebd7ef2087c"
server_key_iv: "4dd9e99cb0be41d0c623db6e0e11e635"
server_key_version: "encryptionkey@0"
I20260927 22:07:51.154306 26551 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 22:07:51.159400 26566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:51.161650 26551 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260927 22:07:51.161866 26551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/wal
uuid: "87b4d93d107e4993ac8c1b12150307e3"
format_stamp: "Formatted at 2026-09-27 22:07:51 on dist-test-slave-sg4j"
server_key: "49a7d69b1455cc8cc745bebd7ef2087c"
server_key_iv: "4dd9e99cb0be41d0c623db6e0e11e635"
server_key_version: "encryptionkey@0"
I20260927 22:07:51.162117 26551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:51.218216 26551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:51.224241 26551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:51.224519 26551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:51.250224 26551 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.62:37619
I20260927 22:07:51.250249 26617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.62:37619 every 8 connection(s)
I20260927 22:07:51.252080 26551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/data/info.pb
I20260927 22:07:51.253983 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 26551
I20260927 22:07:51.254230 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-0/wal/instance
I20260927 22:07:51.254818 3708 external_mini_cluster.cc:1428] Setting key 638dfcb13e7fe6a6ed6f949754d82256
I20260927 22:07:51.258913 26618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:51.272800 26618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.159.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.61@KRBTEST.COM" created.
Sep 27 22:07:51 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546871, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:07:51.316505 26622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.279105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38170 (local address 127.3.159.62:37619)
0927 22:07:51.282146 (+ 3041us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:51.282154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:51.282217 (+ 63us) server_negotiation.cc:407] Connection header received
0927 22:07:51.285579 (+ 3362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:51.285588 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:51.286008 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:51.286245 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:51.288799 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:51.291386 (+ 2587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:51.293746 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:51.294418 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:51.294662 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:51.301114 (+ 6452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:51.301132 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:51.301157 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:51.301222 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:51.308100 (+ 6878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:51.309796 (+ 1696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:51.309813 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:51.309820 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:51.310196 (+ 376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:51.311368 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:51.311377 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:51.311384 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:51.312421 (+ 1037us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:51.312686 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:51.314296 (+ 1610us) server_negotiation.cc:299] Negotiation successful
0927 22:07:51.315021 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2757,"spinlock_wait_cycles":111360,"thread_start_us":2344,"threads_started":1}
I20260927 22:07:51.316606 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.279105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37619 (local address 127.0.0.1:38170)
0927 22:07:51.280554 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.280651 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:51.284685 (+ 4034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:51.286419 (+ 1734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:51.286431 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:51.287602 (+ 1171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:51.288323 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:51.288366 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:51.291675 (+ 3309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:51.291681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:51.293482 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:51.293518 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:51.294716 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:51.294722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:51.294895 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:51.295690 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:51.295718 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:51.300643 (+ 4925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:51.308489 (+ 7846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:51.308507 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:51.308533 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:51.309252 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:51.310761 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:51.310767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:51.310770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:51.311112 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:51.312751 (+ 1639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:51.312767 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:51.313230 (+ 463us) client_negotiation.cc:769] Sending connection context
0927 22:07:51.313754 (+ 524us) client_negotiation.cc:240] Negotiation successful
0927 22:07:51.314740 (+ 986us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":612,"threads_started":1}
I20260927 22:07:51.322106 26618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:07:51.323871 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.323062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:41964)
0927 22:07:51.323392 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.323591 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260927 22:07:51.324309 26570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
I20260927 22:07:51.326596 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.325989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:41970)
0927 22:07:51.326216 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.326360 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 22:07:51.326941 26618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:45999: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab.
Entry for principal kudu/127.3.159.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab.
I20260927 22:07:51.351181 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.61:45999
--webserver_interface=127.3.159.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab
--principal=kudu/127.3.159.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:37619,127.3.159.61:45999,127.3.159.60:38013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:07:51.376468 26618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } attempt: 1
I20260927 22:07:51.378381 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.377370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:41984)
0927 22:07:51.377911 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.378101 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
I20260927 22:07:51.380398 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.379648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:41996)
0927 22:07:51.379894 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.380067 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 22:07:51.380735 26618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:45999: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
I20260927 22:07:51.449101 26618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } attempt: 2
I20260927 22:07:51.450892 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.450155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:42008)
0927 22:07:51.450474 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.450655 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260927 22:07:51.452854 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.452098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:42014)
0927 22:07:51.452343 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.452502 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 22:07:51.453266 26618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:45999: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
I20260927 22:07:51.544659 26618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } attempt: 3
I20260927 22:07:51.546586 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.545915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:42030)
0927 22:07:51.546212 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.546347 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260927 22:07:51.548367 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.547601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:42046)
0927 22:07:51.547871 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.548047 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 22:07:51.548745 26618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:45999: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
I20260927 22:07:51.692080 26618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } attempt: 4
I20260927 22:07:51.693981 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.693081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:42056)
0927 22:07:51.693593 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.693750 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276}
I20260927 22:07:51.695415 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.694843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:42062)
0927 22:07:51.695083 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.695218 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260927 22:07:51.695823 26618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.61:45999: Network error: Client connection negotiation failed: client connection to 127.3.159.61:45999: connect: Connection refused (error 111)
W20260927 22:07:51.806857 26625 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:07:51.807391 26625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:51.807565 26625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:51.807679 26625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:51.817066 26625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:51.817229 26625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:51.817305 26625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:51.817435 26625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:51.817587 26625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:51.833957 26625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:37619,127.3.159.61:45999,127.3.159.60:38013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.61.keytab
--principal=kudu/127.3.159.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.61:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.159.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:51.836361 26625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:51.838634 26625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:51.856603 26630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:51.856678 26633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:51.856591 26631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:51.857704 26625 server_base.cc:1034] running on GCE node
I20260927 22:07:51.858788 26625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:51.860764 26625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:51.862371 26625 hybrid_clock.cc:648] HybridClock initialized: now 1790546871862210 us; error 96 us; skew 500 ppm
I20260927 22:07:51.862892 26625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:51 dist-test-slave-sg4j krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546871, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:51.868451 26625 init.cc:377] Logged in from keytab as kudu/127.3.159.61@KRBTEST.COM (short username kudu)
I20260927 22:07:51.870843 26625 webserver.cc:460] Webserver started at http://127.3.159.61:36671/ using document root <none> and password file <none>
I20260927 22:07:51.871908 26625 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:51.872033 26625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:51.872471 26625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:51.876876 26625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/data/instance:
uuid: "902c86a9409d4574b6b2b698aeff411e"
format_stamp: "Formatted at 2026-09-27 22:07:51 on dist-test-slave-sg4j"
server_key: "0b0f7bb67032905b10db5ba15d82349c"
server_key_iv: "611521dcf9ad338df64fd3fb69ac1f26"
server_key_version: "encryptionkey@0"
I20260927 22:07:51.878103 26625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/wal/instance:
uuid: "902c86a9409d4574b6b2b698aeff411e"
format_stamp: "Formatted at 2026-09-27 22:07:51 on dist-test-slave-sg4j"
server_key: "0b0f7bb67032905b10db5ba15d82349c"
server_key_iv: "611521dcf9ad338df64fd3fb69ac1f26"
server_key_version: "encryptionkey@0"
I20260927 22:07:51.885991 26625 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260927 22:07:51.891366 26640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:51.893725 26625 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260927 22:07:51.894011 26625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/wal
uuid: "902c86a9409d4574b6b2b698aeff411e"
format_stamp: "Formatted at 2026-09-27 22:07:51 on dist-test-slave-sg4j"
server_key: "0b0f7bb67032905b10db5ba15d82349c"
server_key_iv: "611521dcf9ad338df64fd3fb69ac1f26"
server_key_version: "encryptionkey@0"
I20260927 22:07:51.894304 26625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:51.913021 26625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:51.919476 26625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:51.919886 26625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:51.945371 26625 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.61:45999
I20260927 22:07:51.945660 26691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.61:45999 every 8 connection(s)
I20260927 22:07:51.947288 26625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/data/info.pb
I20260927 22:07:51.948319 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 26625
I20260927 22:07:51.948580 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-1/wal/instance
I20260927 22:07:51.949292 3708 external_mini_cluster.cc:1428] Setting key 2125519c5a18ba713af1718b77a81eb6
I20260927 22:07:51.952862 26692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:51.961570 26692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.159.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.159.60@KRBTEST.COM" created.
Sep 27 22:07:51 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546871, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:07:51.994738 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.965148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38184 (local address 127.3.159.62:37619)
0927 22:07:51.965865 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:51.965873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:51.967355 (+ 1482us) server_negotiation.cc:407] Connection header received
0927 22:07:51.971070 (+ 3715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:51.971079 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:51.971220 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:51.971410 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:51.974150 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:51.975017 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:51.977708 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:51.978255 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:51.978439 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:51.984852 (+ 6413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:51.984863 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:51.984872 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:51.984960 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:51.990079 (+ 5119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:51.991498 (+ 1419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:51.991509 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:51.991514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:51.991752 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:51.992375 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:51.992380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:51.992383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:51.992731 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:51.992942 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:51.993993 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0927 22:07:51.994298 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":375,"threads_started":1}
I20260927 22:07:51.995159 26618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } attempt: 5
I20260927 22:07:51.996433 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.965871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37619 (local address 127.0.0.1:38184)
0927 22:07:51.967092 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.967168 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:51.970498 (+ 3330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:51.971753 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:51.971771 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:51.972771 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:51.973795 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:51.973830 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:51.975272 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:51.975278 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:51.977450 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:51.977484 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:51.978802 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:51.978817 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:51.979518 (+ 701us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:51.980637 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:51.980707 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:51.984392 (+ 3685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:51.990386 (+ 5994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:51.990402 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:51.990427 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:51.991237 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:51.991934 (+ 697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:51.991947 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:51.991950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:51.992225 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:51.992893 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:51.992906 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:51.993499 (+ 593us) client_negotiation.cc:769] Sending connection context
0927 22:07:51.993858 (+ 359us) client_negotiation.cc:240] Negotiation successful
0927 22:07:51.994552 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":422,"threads_started":1}
I20260927 22:07:51.998900 26692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:07:52 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546871, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
Sep 27 22:07:52 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546871, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
I20260927 22:07:52.020227 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.999818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:42092)
0927 22:07:52.000280 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.000317 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:52.000442 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:52.002987 (+ 2545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:52.002995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:52.003622 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:52.003917 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.003927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.005339 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.005346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.006688 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.006703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.008008 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.008015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.008230 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.008946 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:52.008971 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:52.011933 (+ 2962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:52.016531 (+ 4598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.016539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.016543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.017065 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.017719 (+ 654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.017726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.017732 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.017917 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.018857 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:52.018861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:52.019092 (+ 231us) client_negotiation.cc:769] Sending connection context
0927 22:07:52.019317 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 22:07:52.019749 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260927 22:07:52.020689 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.995915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:42078)
0927 22:07:51.996163 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:51.996186 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:51.996320 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:51.998280 (+ 1960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:51.998287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:51.998903 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:51.999215 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:51.999256 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.002089 (+ 2833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.002100 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.003566 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.003591 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.005245 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.005252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.005529 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.006432 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:52.006464 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:52.009309 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:52.017009 (+ 7700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.017016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.017020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.017681 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.018345 (+ 664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.018351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.018354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.018579 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.019309 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:52.019316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:52.019548 (+ 232us) client_negotiation.cc:769] Sending connection context
0927 22:07:52.019837 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 22:07:52.020243 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 22:07:52.020807 26699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.000009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.3.159.61:45999)
0927 22:07:52.002551 (+ 2542us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.002559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.002587 (+ 28us) server_negotiation.cc:407] Connection header received
0927 22:07:52.002649 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.002653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.002770 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.002918 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.004102 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.005148 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.006893 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.007715 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.007891 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.012268 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.012274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.012278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.012314 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.016346 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.017272 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.017281 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.017284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.017564 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.018030 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.018038 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.018044 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.018640 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.018880 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.019491 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.020010 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":185,"server-negotiator.queue_time_us":2360,"thread_start_us":2159,"threads_started":1}
I20260927 22:07:52.021320 26698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:51.996614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.3.159.61:45999)
0927 22:07:51.997539 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:51.997550 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:51.997596 (+ 46us) server_negotiation.cc:407] Connection header received
0927 22:07:51.997663 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:51.997670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:51.998027 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:51.998233 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:51.999531 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.001812 (+ 2281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.003796 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.004998 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.005182 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.009935 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.009967 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.010000 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.010067 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.016763 (+ 6696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.017896 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.017902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.017906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.018127 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.018746 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.018754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.018758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.019130 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.019305 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.019960 (+ 655us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.020428 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":1094,"threads_started":1}
I20260927 22:07:52.025112 26618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:07:52.025197 26692 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } has no permanent_uuid. Determining permanent_uuid...
I20260927 22:07:52.026893 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.026132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47720)
0927 22:07:52.026476 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.026636 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260927 22:07:52.026885 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.026192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47734)
0927 22:07:52.026487 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.026685 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260927 22:07:52.027707 26643 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.159.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab.
Entry for principal kudu/127.3.159.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab.
I20260927 22:07:52.028663 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.027921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47742)
0927 22:07:52.028376 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.028497 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292}
W20260927 22:07:52.029177 26618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38013: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
I20260927 22:07:52.029714 3708 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
/tmp/dist-test-taskIPxQZD/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.60:38013
--webserver_interface=127.3.159.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab
--principal=kudu/127.3.159.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.159.20:43925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.159.62:37619,127.3.159.61:45999,127.3.159.60:38013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 22:07:52.030082 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.029557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47752)
0927 22:07:52.029770 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.029866 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:07:52.030361 26692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38013: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
I20260927 22:07:52.079829 26692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } attempt: 1
I20260927 22:07:52.080498 26618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } attempt: 1
I20260927 22:07:52.081598 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.080851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47754)
0927 22:07:52.081204 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.081333 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260927 22:07:52.082515 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.081286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47762)
0927 22:07:52.081648 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.082100 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 22:07:52.083272 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.082771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47772)
0927 22:07:52.082955 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.083091 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:07:52.083550 26692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38013: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
I20260927 22:07:52.084044 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.083591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47784)
0927 22:07:52.083829 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.083918 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260927 22:07:52.084349 26618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38013: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
I20260927 22:07:52.151906 26692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } attempt: 2
I20260927 22:07:52.152619 26618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } attempt: 2
I20260927 22:07:52.153824 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.152868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47788)
0927 22:07:52.153202 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.153344 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260927 22:07:52.154609 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.153604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47800)
0927 22:07:52.153912 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.154359 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 22:07:52.155067 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.154646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47802)
0927 22:07:52.154814 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.154919 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 22:07:52.155371 26692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38013: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
I20260927 22:07:52.155844 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.155431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47814)
0927 22:07:52.155629 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.155718 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260927 22:07:52.156126 26618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38013: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
I20260927 22:07:52.246706 26692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } attempt: 3
I20260927 22:07:52.248716 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.247727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47828)
0927 22:07:52.248127 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.248333 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260927 22:07:52.251127 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.249965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47832)
0927 22:07:52.250668 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.250861 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430}
W20260927 22:07:52.251627 26692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38013: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
I20260927 22:07:52.262418 26618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } attempt: 3
I20260927 22:07:52.264123 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.263327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47840)
0927 22:07:52.263767 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.263891 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
I20260927 22:07:52.265833 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.265206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47846)
0927 22:07:52.265551 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.265683 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260927 22:07:52.266088 26618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38013: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
I20260927 22:07:52.395062 26692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } attempt: 4
I20260927 22:07:52.396734 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.396075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47858)
0927 22:07:52.396389 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.396514 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 22:07:52.398247 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.397814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47860)
0927 22:07:52.398011 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.398105 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 22:07:52.398524 26692 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38013: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
I20260927 22:07:52.443439 26618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } attempt: 4
I20260927 22:07:52.445137 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.444404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47872)
0927 22:07:52.444694 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.444840 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 22:07:52.447005 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.446402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47878)
0927 22:07:52.446662 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.446788 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 22:07:52.447369 26618 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.159.60:38013: Network error: Client connection negotiation failed: client connection to 127.3.159.60:38013: connect: Connection refused (error 111)
W20260927 22:07:52.475813 26701 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 22:07:52.476284 26701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 22:07:52.476426 26701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 22:07:52.476572 26701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 22:07:52.485123 26701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 22:07:52.485266 26701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 22:07:52.485330 26701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 22:07:52.485477 26701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 22:07:52.485571 26701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 22:07:52.497221 26701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.159.20:43925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.159.62:37619,127.3.159.61:45999,127.3.159.60:38013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/krb5kdc/kudu_127.3.159.60.keytab
--principal=kudu/127.3.159.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.159.60:38013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.159.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 20:49:16 UTC on ccycloud.quasar-pnojql.root.comops.site
build id 4676317
ASAN enabled
I20260927 22:07:52.499104 26701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 22:07:52.500873 26701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 22:07:52.516757 26706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:52.516781 26707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 22:07:52.517199 26709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 22:07:52.517699 26701 server_base.cc:1034] running on GCE node
I20260927 22:07:52.518918 26701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 22:07:52.520900 26701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 22:07:52.522301 26701 hybrid_clock.cc:648] HybridClock initialized: now 1790546872522151 us; error 118 us; skew 500 ppm
I20260927 22:07:52.522755 26701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 22:07:52 dist-test-slave-sg4j krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546872, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 22:07:52.527200 26701 init.cc:377] Logged in from keytab as kudu/127.3.159.60@KRBTEST.COM (short username kudu)
I20260927 22:07:52.529619 26701 webserver.cc:460] Webserver started at http://127.3.159.60:32801/ using document root <none> and password file <none>
I20260927 22:07:52.530623 26701 fs_manager.cc:359] Metadata directory not provided
I20260927 22:07:52.530809 26701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 22:07:52.531234 26701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 22:07:52.535323 26701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/data/instance:
uuid: "4b612352178a456c8dd12d5f7951982e"
format_stamp: "Formatted at 2026-09-27 22:07:52 on dist-test-slave-sg4j"
server_key: "ca49f1930a3a4f1be8d487d5adecffbb"
server_key_iv: "be0df43d4c80c0c587847c2e8d4db00a"
server_key_version: "encryptionkey@0"
I20260927 22:07:52.536350 26701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/wal/instance:
uuid: "4b612352178a456c8dd12d5f7951982e"
format_stamp: "Formatted at 2026-09-27 22:07:52 on dist-test-slave-sg4j"
server_key: "ca49f1930a3a4f1be8d487d5adecffbb"
server_key_iv: "be0df43d4c80c0c587847c2e8d4db00a"
server_key_version: "encryptionkey@0"
I20260927 22:07:52.543744 26701 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260927 22:07:52.548689 26716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 22:07:52.551035 26701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260927 22:07:52.551316 26701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/data,/tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/wal
uuid: "4b612352178a456c8dd12d5f7951982e"
format_stamp: "Formatted at 2026-09-27 22:07:52 on dist-test-slave-sg4j"
server_key: "ca49f1930a3a4f1be8d487d5adecffbb"
server_key_iv: "be0df43d4c80c0c587847c2e8d4db00a"
server_key_version: "encryptionkey@0"
I20260927 22:07:52.551646 26701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 22:07:52.569504 26701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 22:07:52.575347 26701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 22:07:52.575717 26701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 22:07:52.602000 26701 rpc_server.cc:307] RPC server started. Bound to: 127.3.159.60:38013
I20260927 22:07:52.602023 26767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.159.60:38013 every 8 connection(s)
I20260927 22:07:52.603869 26701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/data/info.pb
I20260927 22:07:52.605618 3708 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu as pid 26701
I20260927 22:07:52.605851 3708 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIPxQZD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790546447244210-3708-0/minicluster-data/master-2/wal/instance
I20260927 22:07:52.606389 3708 external_mini_cluster.cc:1428] Setting key e063dbb920106531c2feadff87c6d591
I20260927 22:07:52.610586 26768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 22:07:52.620509 26768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:07:52 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
Sep 27 22:07:52 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546872, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
I20260927 22:07:52.639183 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.609757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.3.159.60:38013)
0927 22:07:52.610861 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.610877 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.610929 (+ 52us) server_negotiation.cc:407] Connection header received
0927 22:07:52.613029 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.613066 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.613751 (+ 685us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.614344 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.616886 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.618788 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.620370 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.621311 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.621621 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.628318 (+ 6697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.628339 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.628363 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.628420 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.633178 (+ 4758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.634350 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.634362 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.634372 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.634595 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.635078 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.635081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.635083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.635754 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.636124 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.636758 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.637562 (+ 804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":499,"threads_started":1}
I20260927 22:07:52.644357 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.624375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.62:37619 (local address 127.0.0.1:38186)
0927 22:07:52.625886 (+ 1511us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.625954 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:52.626246 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:52.626906 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:52.626915 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:52.627901 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:52.628498 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.628520 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.630172 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.630177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.631849 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.631868 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.633127 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.633133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.633365 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.634255 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:52.634294 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:52.636927 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:52.640787 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.640806 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.640822 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.641512 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.641933 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.641939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.641942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.642161 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.642899 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:52.642907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:52.643196 (+ 289us) client_negotiation.cc:769] Sending connection context
0927 22:07:52.643429 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 22:07:52.643882 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"thread_start_us":992,"threads_started":1}
I20260927 22:07:52.644767 26772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.624371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.3.159.62:37619)
0927 22:07:52.625307 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.625317 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.626130 (+ 813us) server_negotiation.cc:407] Connection header received
0927 22:07:52.626446 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.626452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.626609 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.626765 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.628726 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.629937 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.632069 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.632830 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.633020 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.637219 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.637225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.637228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.637278 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.640577 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.641659 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.641666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.641674 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.641810 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.642348 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.642352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.642354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.642710 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.642898 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.643925 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.644225 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":552,"threads_started":1}
I20260927 22:07:52.646706 26768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:07:52 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.62@KRBTEST.COM
Sep 27 22:07:52 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546872, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
I20260927 22:07:52.665779 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.643186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.3.159.62:37619)
0927 22:07:52.643895 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.643899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.643920 (+ 21us) server_negotiation.cc:407] Connection header received
0927 22:07:52.643964 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.643967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.644044 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.644288 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.646861 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.647994 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.650091 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.650592 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.650748 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.656750 (+ 6002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.656759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.656771 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.656844 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.661686 (+ 4842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.663032 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.663039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.663042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.663290 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.663770 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.663775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.663780 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.664182 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.664553 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.664927 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.665169 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":356,"threads_started":1}
I20260927 22:07:52.666096 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.647596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.61:45999 (local address 127.0.0.1:42102)
0927 22:07:52.647820 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.647844 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:52.647980 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:52.648690 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:52.648699 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:52.649273 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:52.649530 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.649541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.651145 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.651153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.652685 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.652703 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.653704 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.653710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.653976 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.654756 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:52.654776 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:52.657935 (+ 3159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:52.662621 (+ 4686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.662628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.662633 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.663341 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.663874 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.663877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.663880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.664056 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.664822 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:52.664830 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:52.665047 (+ 217us) client_negotiation.cc:769] Sending connection context
0927 22:07:52.665243 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 22:07:52.665655 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 22:07:52.666105 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.647782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.3.159.61:45999)
0927 22:07:52.648383 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.648388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.648405 (+ 17us) server_negotiation.cc:407] Connection header received
0927 22:07:52.648455 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.648458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.648520 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.648614 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.649845 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.650972 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.652922 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.653506 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.653675 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.658415 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.658427 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.658436 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.658497 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.662391 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.663522 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.663527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.663531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.663713 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.664199 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.664203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.664208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.664629 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.664791 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.665413 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.665697 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":316,"threads_started":1}
I20260927 22:07:52.667502 26768 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 22:07:52 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546872, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.60@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
Sep 27 22:07:52 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.159.61@KRBTEST.COM
I20260927 22:07:52.684968 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.668773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47896 (local address 127.3.159.60:38013)
0927 22:07:52.668960 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.668968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.669156 (+ 188us) server_negotiation.cc:407] Connection header received
0927 22:07:52.669353 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.669359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.669490 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.669601 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.670418 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.671168 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.672951 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.673645 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.673795 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.678420 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.678426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.678430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.678467 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.681956 (+ 3489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.682758 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.682765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.682769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.682909 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.683306 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.683314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.683318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.683655 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.683815 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.684203 (+ 388us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.684487 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 22:07:52.685065 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.668715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47896)
0927 22:07:52.669023 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.669052 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:52.669206 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:52.669658 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:52.669663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:52.670124 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:52.670272 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.670281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.671311 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.671318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.672714 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.672748 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.673848 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.673854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.674100 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.675034 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:52.675059 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:52.677969 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:52.682159 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.682165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.682170 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.682619 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.683042 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.683047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.683050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.683194 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.683785 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:52.683790 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:52.683946 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 22:07:52.684114 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 22:07:52.684669 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":172}
I20260927 22:07:52.686321 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.668303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.3.159.61:45999)
0927 22:07:52.668519 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.668524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.668990 (+ 466us) server_negotiation.cc:407] Connection header received
0927 22:07:52.669067 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.669071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.669181 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.669280 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.670829 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.671653 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.673094 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.673735 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.673886 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.679625 (+ 5739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.679631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.679635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.679684 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.682979 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.683811 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.683818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.683821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.683974 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.684373 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.684379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.684382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.684693 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.684993 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.685724 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.686036 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 22:07:52.692634 26768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e: Bootstrap starting.
I20260927 22:07:52.693584 3708 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 22:07:52.696354 26768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:52.697813 26692 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } attempt: 5
I20260927 22:07:52.698654 26768 log.cc:824] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:52.696641 26593 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "87b4d93d107e4993ac8c1b12150307e3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38196
I20260927 22:07:52.703578 26768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e: No bootstrap required, opened a new log
Sep 27 22:07:52 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546871, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.61@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
I20260927 22:07:52.717584 26768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } }
I20260927 22:07:52.718133 26768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b612352178a456c8dd12d5f7951982e, State: Initialized, Role: FOLLOWER
I20260927 22:07:52.718817 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.698802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47900)
0927 22:07:52.699175 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.699201 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:52.699422 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:52.699824 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:52.699830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:52.700533 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:52.700807 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.700826 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.702770 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.702780 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.704398 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.704421 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.705369 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.705433 (+ 64us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.705712 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.706454 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:52.706474 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:52.709263 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:52.714835 (+ 5572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.714843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.714847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.715536 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.716100 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.716105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.716109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.716307 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.717287 (+ 980us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:52.717357 (+ 70us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:52.717741 (+ 384us) client_negotiation.cc:769] Sending connection context
0927 22:07:52.717947 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 22:07:52.718342 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260927 22:07:52.718820 26768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } }
I20260927 22:07:52.719202 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.698907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.3.159.60:38013)
0927 22:07:52.699203 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.699210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.699311 (+ 101us) server_negotiation.cc:407] Connection header received
0927 22:07:52.699576 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.699582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.699682 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.699804 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.701067 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.702483 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.704603 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.705162 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.705329 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.709609 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.709621 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.709632 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.709702 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.714504 (+ 4802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.715736 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.715742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.715746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.715940 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.716454 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.716459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.716462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.716944 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.717140 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.718443 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.718728 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260927 22:07:52.720253 26777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } } }
I20260927 22:07:52.720763 26777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:52.721988 26768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:52.724679 26618 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } attempt: 5
I20260927 22:07:52.727523 26692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e: Bootstrap starting.
I20260927 22:07:52.732898 26692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:52.734848 26692 log.cc:824] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e: Log is configured to *not* fsync() on all Append() calls
Sep 27 22:07:52 dist-test-slave-sg4j krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790546871, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.159.62@KRBTEST.COM for kudu/127.3.159.60@KRBTEST.COM
I20260927 22:07:52.747807 26692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e: No bootstrap required, opened a new log
W20260927 22:07:52.749286 26788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:07:52.749488 26788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:07:52.732975 26701 master_runner.cc:186] Error getting master registration for 127.3.159.62:37619: OK, instance_id { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" instance_seqno: 1790546871242559 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:07:52.751376 26701 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:07:52.758505 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.726052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.3.159.60:38013)
0927 22:07:52.735833 (+ 9781us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.735842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.735875 (+ 33us) server_negotiation.cc:407] Connection header received
0927 22:07:52.735963 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.735970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.736089 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.737638 (+ 1549us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.737837 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.739139 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.741828 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.742677 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.743715 (+ 1038us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.749155 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.749164 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.749169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.749226 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.754333 (+ 5107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.755518 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.755525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.755530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.755759 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.756475 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.756481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.756485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.756965 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.757151 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.757702 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.758044 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9567}
I20260927 22:07:52.758517 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.725598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.159.60:38013 (local address 127.0.0.1:47910)
0927 22:07:52.726113 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 22:07:52.726153 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 22:07:52.726319 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 22:07:52.736400 (+ 10081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 22:07:52.736420 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 22:07:52.737049 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0927 22:07:52.737360 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.737431 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.739392 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.739400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.741485 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 22:07:52.741531 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.742930 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.742938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 22:07:52.743219 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.744792 (+ 1573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 22:07:52.744842 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0927 22:07:52.748776 (+ 3934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 22:07:52.754632 (+ 5856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.754639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.754644 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.755293 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.756011 (+ 718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 22:07:52.756016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 22:07:52.756019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 22:07:52.756290 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 22:07:52.757108 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 22:07:52.757113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 22:07:52.757323 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 22:07:52.757526 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 22:07:52.758069 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319}
I20260927 22:07:52.762916 26692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } }
I20260927 22:07:52.763406 26692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902c86a9409d4574b6b2b698aeff411e, State: Initialized, Role: FOLLOWER
I20260927 22:07:52.764034 26692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } }
I20260927 22:07:52.766006 26791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } } }
I20260927 22:07:52.767385 26791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:52.767656 26618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3: Bootstrap starting.
I20260927 22:07:52.767972 26692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:52.773614 26618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3: Neither blocks nor log segments found. Creating new log.
I20260927 22:07:52.775656 26618 log.cc:824] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3: Log is configured to *not* fsync() on all Append() calls
I20260927 22:07:52.781625 26618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3: No bootstrap required, opened a new log
I20260927 22:07:52.790817 26618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } }
I20260927 22:07:52.791419 26618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b4d93d107e4993ac8c1b12150307e3, State: Initialized, Role: FOLLOWER
I20260927 22:07:52.792088 26618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } }
W20260927 22:07:52.793928 26803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 22:07:52.794202 26803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 22:07:52.795106 26805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } } }
I20260927 22:07:52.795822 26805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:52.797992 26618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 22:07:52.798038 26593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 22:07:52.798522 26593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:52.789587 26625 master_runner.cc:186] Error getting master registration for 127.3.159.62:37619: OK, instance_id { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" instance_seqno: 1790546871242559 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 22:07:52.803251 26625 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 22:07:52.803212 26593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } }
I20260927 22:07:52.805639 26593 leader_election.cc:288] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [CANDIDATE]: Term 1 election: Requested vote from peers 902c86a9409d4574b6b2b698aeff411e (127.3.159.61:45999), 4b612352178a456c8dd12d5f7951982e (127.3.159.60:38013)
I20260927 22:07:52.806789 26667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b4d93d107e4993ac8c1b12150307e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "902c86a9409d4574b6b2b698aeff411e"
I20260927 22:07:52.806782 26743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87b4d93d107e4993ac8c1b12150307e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4b612352178a456c8dd12d5f7951982e"
I20260927 22:07:52.807549 26667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:52.807556 26743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [term 0 FOLLOWER]: Advancing to term 1
I20260927 22:07:52.811460 26743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b4d93d107e4993ac8c1b12150307e3 in term 1.
I20260927 22:07:52.811452 26667 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b4d93d107e4993ac8c1b12150307e3 in term 1.
I20260927 22:07:52.812381 26569 leader_election.cc:302] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4b612352178a456c8dd12d5f7951982e, 87b4d93d107e4993ac8c1b12150307e3, 902c86a9409d4574b6b2b698aeff411e; no voters:
I20260927 22:07:52.812902 26805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 22:07:52.814035 26805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [term 1 LEADER]: Becoming Leader. State: Replica: 87b4d93d107e4993ac8c1b12150307e3, State: Running, Role: LEADER
I20260927 22:07:52.815001 26805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } }
W20260927 22:07:52.832633 26722 tablet.cc:2369] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:07:52.866287 26809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b4d93d107e4993ac8c1b12150307e3. Latest consensus state: current_term: 1 leader_uuid: "87b4d93d107e4993ac8c1b12150307e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } } }
I20260927 22:07:52.866905 26809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:52.880149 26774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.820410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.3.159.62:37619)
0927 22:07:52.820678 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.820690 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.840198 (+ 19508us) server_negotiation.cc:407] Connection header received
0927 22:07:52.844808 (+ 4610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.844820 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.844947 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.845043 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.860147 (+ 15104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.861752 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.866174 (+ 4422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.866978 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.867125 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.871212 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.871223 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.871231 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.871296 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.875934 (+ 4638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.877069 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.877079 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.877084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.877320 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.877906 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.877915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.877922 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.878389 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.878675 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.879391 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.879683 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 22:07:52.883167 26826 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 22:07:52.890751 26824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 22:07:52.915483 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.817333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47918 (local address 127.3.159.60:38013)
0927 22:07:52.817597 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.817604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.836909 (+ 19305us) server_negotiation.cc:407] Connection header received
0927 22:07:52.841247 (+ 4338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.841257 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.841438 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.841552 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.851772 (+ 10220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.853284 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.867455 (+ 14171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.868335 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.868500 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.891496 (+ 22996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.891504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.891509 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.891559 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.906972 (+ 15413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.908415 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.908423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.908454 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.908818 (+ 364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.909943 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.909979 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.909983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.910719 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.913044 (+ 2325us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.913435 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.913974 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260927 22:07:52.923727 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:52.837781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.3.159.61:45999)
0927 22:07:52.838043 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:52.838052 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:52.841063 (+ 3011us) server_negotiation.cc:407] Connection header received
0927 22:07:52.845335 (+ 4272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:52.845344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:52.845507 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:52.845611 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:52.855747 (+ 10136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.857283 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.869058 (+ 11775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:52.869912 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:52.870096 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:52.884613 (+ 14517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 22:07:52.884619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 22:07:52.884622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 22:07:52.884667 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 22:07:52.888569 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.918645 (+ 30076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.918657 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.918662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.919051 (+ 389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 22:07:52.920694 (+ 1643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 22:07:52.920702 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 22:07:52.920706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 22:07:52.921340 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 22:07:52.921741 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:52.922758 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0927 22:07:52.923235 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260927 22:07:52.923724 26646 tablet.cc:2369] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:07:52.943629 26824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 22:07:52.978804 26572 tablet.cc:2369] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 22:07:52.993441 26743 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [term 1 FOLLOWER]: Refusing update from remote peer 87b4d93d107e4993ac8c1b12150307e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:07:52.995184 26809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:07:53.009807 26667 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [term 1 FOLLOWER]: Refusing update from remote peer 87b4d93d107e4993ac8c1b12150307e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 22:07:53.015889 26809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 22:07:53.053506 26791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b4d93d107e4993ac8c1b12150307e3. Latest consensus state: current_term: 1 leader_uuid: "87b4d93d107e4993ac8c1b12150307e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } } }
I20260927 22:07:53.053915 26791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:53.056412 26828 mvcc.cc:205] Tried to move back new op lower bound from 7334079991762161664 to 7334079991056515072. Current Snapshot: MvccSnapshot[applied={T|T < 7334079991762161664}]
I20260927 22:07:53.061518 26830 mvcc.cc:205] Tried to move back new op lower bound from 7334079991762161664 to 7334079991056515072. Current Snapshot: MvccSnapshot[applied={T|T < 7334079991762161664}]
I20260927 22:07:53.068235 26791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87b4d93d107e4993ac8c1b12150307e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } } }
I20260927 22:07:53.068640 26791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:53.079398 26777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b4d93d107e4993ac8c1b12150307e3. Latest consensus state: current_term: 1 leader_uuid: "87b4d93d107e4993ac8c1b12150307e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } } }
I20260927 22:07:53.079782 26777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:53.084192 26809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87b4d93d107e4993ac8c1b12150307e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } } }
I20260927 22:07:53.084591 26809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:53.085090 26829 mvcc.cc:205] Tried to move back new op lower bound from 7334079991762161664 to 7334079991056515072. Current Snapshot: MvccSnapshot[applied={T|T < 7334079991762161664}]
I20260927 22:07:53.085101 26809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87b4d93d107e4993ac8c1b12150307e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } } }
I20260927 22:07:53.085358 26809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3 [sys.catalog]: This master's current role is: LEADER
I20260927 22:07:53.086755 26777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87b4d93d107e4993ac8c1b12150307e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87b4d93d107e4993ac8c1b12150307e3" member_type: VOTER last_known_addr { host: "127.3.159.62" port: 37619 } } peers { permanent_uuid: "902c86a9409d4574b6b2b698aeff411e" member_type: VOTER last_known_addr { host: "127.3.159.61" port: 45999 } } peers { permanent_uuid: "4b612352178a456c8dd12d5f7951982e" member_type: VOTER last_known_addr { host: "127.3.159.60" port: 38013 } } }
I20260927 22:07:53.087036 26777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 22:07:53.087419 26824 catalog_manager.cc:1347] Generated new cluster ID: 1d7452aef2654afca34b3be6e7b270ba
I20260927 22:07:53.087527 26824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 22:07:53.128371 26824 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 22:07:53.132201 26824 catalog_manager.cc:1504] Loading token signing keys...
I20260927 22:07:53.156221 26824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b4d93d107e4993ac8c1b12150307e3: Generated new TSK 0
I20260927 22:07:53.158313 26824 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 22:07:53.327286 26771 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:53.324545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.3.159.60:38013)
0927 22:07:53.324793 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.324800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.324840 (+ 40us) server_negotiation.cc:407] Connection header received
0927 22:07:53.325082 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.325090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.325194 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.325299 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:53.326951 (+ 1652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47928: BlockingRecv error: recv got EOF from 127.0.0.1:47928 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260927 22:07:53.330777 26774 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:53.326440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.3.159.62:37619)
0927 22:07:53.326681 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.326688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.328407 (+ 1719us) server_negotiation.cc:407] Connection header received
0927 22:07:53.328750 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.328757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.328889 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.328980 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:53.330427 (+ 1447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38210: BlockingRecv error: recv got EOF from 127.0.0.1:38210 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260927 22:07:53.336694 26775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:53.333192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.3.159.61:45999)
0927 22:07:53.333491 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.333500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.334214 (+ 714us) server_negotiation.cc:407] Connection header received
0927 22:07:53.334565 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.334573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.334691 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.334784 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:53.336307 (+ 1523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42132: BlockingRecv error: recv got EOF from 127.0.0.1:42132 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260927 22:07:53.392702 26775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:53.380715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.3.159.61:45999)
0927 22:07:53.381154 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.381162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.389544 (+ 8382us) server_negotiation.cc:407] Connection header received
0927 22:07:53.390379 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.390388 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.390596 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.390710 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:53.392323 (+ 1613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42138: BlockingRecv error: recv got EOF from 127.0.0.1:42138 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260927 22:07:53.446990 26775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:53.436116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.3.159.61:45999)
0927 22:07:53.436504 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.436513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.444185 (+ 7672us) server_negotiation.cc:407] Connection header received
0927 22:07:53.444920 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.444928 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.445076 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.445173 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:53.446681 (+ 1508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42144: BlockingRecv error: recv got EOF from 127.0.0.1:42144 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
W20260927 22:07:53.498459 26775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:53.487723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.3.159.61:45999)
0927 22:07:53.488031 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.488039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.495355 (+ 7316us) server_negotiation.cc:407] Connection header received
0927 22:07:53.496079 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.496087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.496227 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.496333 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:53.497756 (+ 1423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42148: BlockingRecv error: recv got EOF from 127.0.0.1:42148 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260927 22:07:53.537900 26775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:53.534657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.3.159.61:45999)
0927 22:07:53.535010 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.535018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.535764 (+ 746us) server_negotiation.cc:407] Connection header received
0927 22:07:53.536144 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.536153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.536268 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.536363 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:53.537607 (+ 1244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42154: BlockingRecv error: recv got EOF from 127.0.0.1:42154 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
W20260927 22:07:53.554708 26775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:53.551940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.3.159.61:45999)
0927 22:07:53.552188 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.552196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.552586 (+ 390us) server_negotiation.cc:407] Connection header received
0927 22:07:53.553013 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.553020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.553166 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.553337 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:53.554389 (+ 1052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42170: BlockingRecv error: recv got EOF from 127.0.0.1:42170 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260927 22:07:53.578115 26775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:53.575399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42174 (local address 127.3.159.61:45999)
0927 22:07:53.575667 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.575675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.576055 (+ 380us) server_negotiation.cc:407] Connection header received
0927 22:07:53.576242 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.576252 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.576395 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.576577 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:53.577805 (+ 1228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42174: BlockingRecv error: recv got EOF from 127.0.0.1:42174 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260927 22:07:53.627250 26775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:53.624273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.3.159.61:45999)
0927 22:07:53.624496 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.624503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.625299 (+ 796us) server_negotiation.cc:407] Connection header received
0927 22:07:53.625545 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.625552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.625698 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.625873 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:53.626915 (+ 1042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42176: BlockingRecv error: recv got EOF from 127.0.0.1:42176 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 22:07:53.699445 26775 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 22:07:53.695421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.3.159.61:45999)
0927 22:07:53.695769 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.695780 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.696457 (+ 677us) server_negotiation.cc:407] Connection header received
0927 22:07:53.696962 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.696968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.697088 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.697509 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 22:07:53.699097 (+ 1588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42190: BlockingRecv error: recv got EOF from 127.0.0.1:42190 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
I20260927 22:07:53.753180 26788 catalog_manager.cc:1259] Loaded cluster ID: 1d7452aef2654afca34b3be6e7b270ba
I20260927 22:07:53.753553 26788 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e: loading cluster ID for follower catalog manager: success
I20260927 22:07:53.756657 26788 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e: acquiring CA information for follower catalog manager: success
I20260927 22:07:53.758981 26788 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b612352178a456c8dd12d5f7951982e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:07:53.797216 26803 catalog_manager.cc:1259] Loaded cluster ID: 1d7452aef2654afca34b3be6e7b270ba
I20260927 22:07:53.797484 26803 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e: loading cluster ID for follower catalog manager: success
I20260927 22:07:53.801137 26803 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e: acquiring CA information for follower catalog manager: success
I20260927 22:07:53.804113 26803 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 902c86a9409d4574b6b2b698aeff411e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 22:07:53.846223 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:53.833048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42206 (local address 127.3.159.61:45999)
0927 22:07:53.833313 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.833322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.834173 (+ 851us) server_negotiation.cc:407] Connection header received
0927 22:07:53.834694 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.834701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.835078 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.835213 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:07:53.836082 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:53.837751 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:53.841189 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:53.842356 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:53.842834 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:53.843905 (+ 1071us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:07:53.845005 (+ 1100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:07:53.845267 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:53.845603 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 22:07:53.845760 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260927 22:07:53.864678 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:53.855154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.3.159.60:38013)
0927 22:07:53.855859 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.855866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.856190 (+ 324us) server_negotiation.cc:407] Connection header received
0927 22:07:53.856536 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.856542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.856789 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.857013 (+ 224us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:07:53.858011 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:53.858793 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:53.860533 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:53.861034 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:53.861210 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:53.861955 (+ 745us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:07:53.863070 (+ 1115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:07:53.863367 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:53.864127 (+ 760us) server_negotiation.cc:299] Negotiation successful
0927 22:07:53.864268 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":426,"threads_started":1}
I20260927 22:07:53.891834 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:53.876636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.3.159.60:38013)
0927 22:07:53.877361 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.877371 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.882315 (+ 4944us) server_negotiation.cc:407] Connection header received
0927 22:07:53.882755 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.882762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.882914 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.883009 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:07:53.884152 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:53.885004 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:53.888914 (+ 3910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:53.889768 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:53.889963 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:53.890627 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:07:53.890853 (+ 226us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:07:53.890971 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:53.891281 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 22:07:53.891419 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476}
I20260927 22:07:53.892081 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:53.876393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.3.159.62:37619)
0927 22:07:53.877135 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.877147 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.878231 (+ 1084us) server_negotiation.cc:407] Connection header received
0927 22:07:53.879275 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.879286 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.879935 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.880392 (+ 457us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:07:53.882329 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:53.883984 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:53.888769 (+ 4785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:53.889333 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:53.889596 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:53.890515 (+ 919us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:07:53.891253 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:07:53.891371 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:53.891608 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 22:07:53.891729 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":437,"threads_started":1}
I20260927 22:07:53.894585 26775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 22:07:53.878827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42216 (local address 127.3.159.61:45999)
0927 22:07:53.879146 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 22:07:53.879155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 22:07:53.883557 (+ 4402us) server_negotiation.cc:407] Connection header received
0927 22:07:53.883948 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 22:07:53.883955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 22:07:53.884105 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 22:07:53.884561 (+ 456us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 22:07:53.886095 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:53.887471 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:53.890859 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 22:07:53.891812 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 22:07:53.892015 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 22:07:53.893618 (+ 1603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 22:07:53.893802 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 22:07:53.893941 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 22:07:53.894157 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 22:07:53.894271 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 22:07:53.908798 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 26551
I20260927 22:07:53.910231 26613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:54.052440 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 26551
I20260927 22:07:54.082938 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 26625
I20260927 22:07:54.084899 26687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:54.235360 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 26625
I20260927 22:07:54.261826 3708 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu pid 26701
I20260927 22:07:54.263455 26763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 22:07:54.395328 3708 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIPxQZD/build/asan/bin/kudu with pid 26701
2026-09-27T22:07:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5359 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (11207 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (427111 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST